Data independent relevance evaluation utilizing cognitive concept relationship

• US 7,660,786 B2
• Filed: 12/14/2005
• Issued: 02/09/2010
• Est. Priority Date: 12/14/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Queva, Pascale, Li, Ying, Wang, Lee
• Primary Examiner(s): Cottingham; John R.
• Assistant Examiner(s): Badawi; Sherief
• Application Number:
  US11/304,450
• Publication Number:
  US 20070136343A1
• Time in Patent Office: 1,518 Days
• Field of Search: 707/2, 707/3, 707/7, 707/102, 707/100, 707/5, 709/224, 701/02, 705/14
• US Class Current: 1/1
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/08 Auctions
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A measurable means to evaluate a given relevancy verification process to a human decision process is provided. For example, a cognitive concept relationship (CCR) system can be utilized to provide a relevancy verification evaluation that is independent of the relevancy quality of the test data employed to train the relevancy verification algorithm under test. This provides a means to evaluate relevancy verification of, for example, keyword/item pairs for any number of business applications with different relevance standards and/or changes in relevance standards over time, without the need to manually re-label the test data and/or re-measure the algorithmic relevancy.

Claims

1. A system that facilitates relevancy verification evaluation of data item pairs for business applications, the system comprising:

a memory for storing data; and

a processor coupled to the memory configured to act as;

a receiving component that obtains an input data item pair for relevancy evaluation; and

an evaluation component that employs an evaluation system to classify the input data item pair into a cognitive concept relationship (CCR) type indicating a determined relationship between respective input data items within the input data item pair, the CCR type being selected from a set of CCR types based on the determined relationship between respective input data items within the input data item pair, to obtain a mapping relationship between business relevancy decisions and respective CCR types based on a business relevancy standard, and to facilitate a relevancy evaluation of the input data item pair by mapping the selected CCR type into which the input data item pair is classified to a business relevancy decision corresponding to the selected CCR type based on the obtained mapping relationship, wherein the relevancy evaluation is conducted independent of the input data item pair.

2. The system of claim 1, the input data item pair is associated with an advertising keyword and one of an advertising target or an advertising entity.

3. The system of claim 1, the evaluation component utilizes a cognitive concept relationship (CCR) system to facilitate classification of the input data pair.

4. The system of claim 3, the CCR system employs human classification of test cases.

5. The system of claim 3, the evaluation component provides a measurement of algorithmic relevancy accuracy.

6. The system of claim 1 further comprising:

a business model component that performs a relevancy evaluation based on the set of cognitive CCR types to adaptively determine relevance of a keyword in relation to a business model, wherein the business model maps each CCR type to a relevancy decision that specifies the keyword to be one of relevant or non-relevant.

7. The system of claim 6 further comprising:

an advertising component that determines a business advertising decision for the keyword based on the relevancy evaluation performed by the business model component.

8. The system of claim 1, the evaluation component supports multiple business relevancy standards concurrently without requiring retraining of the evaluation component for the respective business relevancy standards.

9. The system of claim 1, wherein the system is implemented in an advertising keyword auction system.

10. A method for facilitating relevancy verification evaluation of a data item pair, comprising:

employing a processor executing computer-executable instructions stored on a computer-readable storage medium to implement the following acts;

obtaining an input data item pair to be evaluated for relevance;

utilizing a cognitive concept relationship (CCR) system to associate the input data item pair with one or more CCR types that indicate a determined relationship between respective input data items within the input data item pair associated with the CCR system based on the determined relationship between respective data items of the input data item pair;

applying a first business relevancy standard to map the CCR types associated with the CCR system to a first set of respective business relevancy decisions independent of the input data item pair and algorithmic relevance verification training data;

applying a second business relevancy standard disparate from the first business relevancy standard to map the CCR types associated with the CCR system to a second set of respective business relevancy decisions independent of the input data item pair and algorithmic relevance verification training data; and

evaluating relevancy of the input data item pair according to the first business relevancy standard and the second business relevancy standard by determining at least one business relevancy decision from the first set of business relevancy decisions and at least one business relevancy decision from the second set of business relevancy decisions that are mapped to the one or more CCR types associated with the input data item pair independently of the respective data items of the input data item pair, the business relevancy decisions specify the input data item pair to be one of relevant or non-relevant based upon the CCR types.

11. The method of claim 10, the input data item pair comprises a first data item associated with an advertising keyword and a second data item associated with one or more of an advertising entity or a target.

12. The method of claim 10 further comprising:

employing test cases for the CCR system that are classified by humans and categorizing the test cases by CCR type.

13. The method of claim 10 further comprising:

utilizing the CCR system to measure algorithmic accuracy for relevance.

14. The method of claim 10 further comprising:

mapping the CCR types associated with the CCR system to one or more business relevancy standards directly based on one or more of business applications with different business relevancy standards, changes in a business relevancy standard over time, or concurrent multiple business relevancy standards.

15. An online Internet auction method that employs the method of claim 10 to facilitate automatic auctioning of keywords.

16. The system of claim 10, wherein the evaluation component adapts to changes in a business relevancy standard by altering the obtained mapping relationship between business relevancy decisions and respective cognitive types over time without requiring retraining of the evaluation component.

17. The method of claim 10, wherein the evaluating relevancy comprises:

utilizing one or more business relevancy models to evaluate the one or more CCR types associated with the input data item pair; and

obtaining a relevancy determination from the one or more business relevancy models based at least in part on the evaluation.

18. A computer-readable storage medium having stored thereon computer-executable instructions to implement acts that facilitate generation of a relevancy decisions in search engine keyword auctions, the acts comprising:

obtaining a pairing of a keyword and an advertising target;

employing a cognitive concept relationship system to the pairing to identify a relationship between the keyword and the advertising target;

assigning a cognitive concept relationship type indicating the identified relationship between the keyword and the advertising target to the pairing in accordance with the identified relationship, the assigned type indicates one of;

the keyword and the advertising target are identical;

the keyword is a superset of the advertising target;

the keyword is a subset of the advertising target;

the keyword and the advertising target overlap;

or

the keyword and the advertising target are respective members of disjoint cognitive concept categories;

utilizing a relevancy model to generate a keyword relevancy based upon the cognitive concept relationship type assigned to the pairing, the relevancy model can be modular such that relevancy models can be interchanged to generate distinct keyword relevancies without alteration of the cognitive concept relationship system; and

generating a decision based upon the keyword relevancy provided by the relevancy model, the decision is one of a decision to bid on the keyword to be associated with the advertising target or a decision to not bid on the keyword to be associated with the advertising target.