MIXING KNOWLEDGE SOURCES WITH AUTO LEARNING FOR IMPROVED ENTITY EXTRACTION

• US 20110022550A1
• Filed: 07/24/2009
• Published: 01/27/2011
• Est. Priority Date: 07/24/2009
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Pantel, Patrick, Pennacchiotti, Marco
• Granted Patent: US 8,499,008 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

The disclosed embodiments of computer systems and techniques utilize an ensemble semantics framework to combine knowledge acquisition systems that yield significantly higher quality resources than each system in isolation. Gains in entity extraction are achieved by combining state-of-the-art distributional and pattern-based systems with a large set of features from, for example, a webcrawl, query logs, and wisdom of the crowd sources. This results in improved query interpretation and greater relevancy in providing search results and advertising, for example.

Claims

1. A method of providing information to users, the computer system configured to:

extracting instances from a plurality of sources using a plurality of knowledge extractors;

aggregating the instances from the plurality of sources;

extracting a plurality of feature vectors for a plurality of instances using a plurality of feature generators, each feature vector extracted by a plurality of feature generators; and

building a model by using a modeler and utilizing the feature vectors extracted by the plurality of feature generators and extracted instances,

wherein building the model comprises automatically generating training sets of instances.

2. The method of claim 1, wherein building the model further comprises utilizing trusted positive examples in generating the training sets.

3. The method of claim 2, further comprising automatically generating the trusted positive instances with a trusted knowledge extractor of the plurality of knowledge extractors.

4. The method of claim 1, wherein building the decoder further comprises utilizing external positive instances in generating the training sets.

5. The method of claim 1, wherein building the decoder further comprises utilizing same class negative instances in generating the training sets.

6. The method of claim 1, wherein building the decoder further comprises utilizing near class negative instances in generating the training sets.

7. The method of claim 5, further comprising acquiring the same class negatives by taking a random sample of instances extracted by only a distributional knowledge extractor of the plurality of knowledge extractors.

8. The method of claim 7, further comprising acquiring the same class negatives by taking a random sample of instances extracted by only a pattern based knowledge extractor of the plurality of knowledge extractors.

9. The method of claim 1, wherein building the decoder further comprises utilizing generic negative instances in generating the training sets.

10. A computer system for providing results to users, the computer system configured to:

extract instances from a plurality of sources using a plurality of knowledge extractors;

aggregate the instances;

extract a plurality of feature vectors using a plurality of feature generators, wherein one of the feature generators

extracts contexts of a query log for a plurality of seeds;

calculates an association statistic between the contexts and seeds;

sorts the contexts by the calculated association statistics and selects a group of the sorted contexts;

for each selected context, generates a feature for a candidate instance comprising the association statistic between the candidate instance and the context; and

wherein the computer system is configured to automatically build labeled training sets from extracted feature vectors and one or more sources of negative instances.

11. The computer system of claim 10, wherein the computer system is configured to automatically generate the training sets with trusted positive instances.

12. The computer system of claim 11, wherein the computer system is configured to generate the trusted positive examples with a trusted knowledge extractor of the plurality of knowledge extractors.

13. The computer system of claim 10, wherein the computer system is configured to automatically generate the training sets with external positive instances.

14. The computer system of claim 10, wherein the computer system is configured to automatically generate the training sets with same class negative instances.

15. The computer system of claim 14, wherein the computer system is configured to automatically generate the training sets with same class negatives acquired as a random sample of instances extracted by only a distributional knowledge extractor of the plurality of knowledge extractors.

16. The computer system of claim 14, wherein the computer system is configured to automatically generate the training sets with same class negatives acquired as a random sample of instances extracted by only a pattern based knowledge extractor of the plurality of knowledge extractors.

17. The computer system of claim 10, wherein the computer system is configured to automatically generate the training sets with generic negative instances.