METHODS, SYSTEMS, AND MEDIA FOR PROVIDING DIRECT AND HYBRID DATA ACQUISITION APPROACHES

• US 20110173037A1
• Filed: 01/06/2011
• Published: 07/14/2011
• Est. Priority Date: 01/06/2010
• Status: Active Grant

• Current Assignee: Integral Ad Science Incorporated (Integral Ad Science Holding Corp.)
• Original Assignee: Integral Ad Science Incorporated (Integral Ad Science Holding Corp.)
• Inventors: Attenberg, Joshua M., Provost, Foster J.
• Granted Patent: US 11,403,568 B2
• Time in Patent Office: Days
• US Class Current: 705/7.11
• CPC Class Codes: G06Q 10/063 Operations research, analysis or management
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

Methods, systems, and media for providing direct and hybrid data acquisition approaches are provided. In accordance with some embodiments of the disclosed subject matter, a method of data acquisition for construction of classification models that incorporates multiple human reviewing resources is provided, the method comprising: receiving a cost structure for constructing a classification model using a data set; instructing a plurality of human reviewing resources to search through the data set and select one or more instances of a class that satisfy at least one criterion, wherein the plurality of human reviewing resources are provided with a definition of the class; training the classification model with the one or more instances from the plurality of human reviewing resources; determining when an expected gain for performing additional searches by the plurality of human reviewing resources as a function of the cost structure is lower than a given threshold; and, in response to determining that the expected gain as a function of the cost structure is lower than the given threshold, instructing the plurality of human reviewing resources that was searching through the data set to label one or more examples from the data set.

Claims

1. A method for data acquisition for construction of classification models that incorporates multiple human reviewing resources, the method comprising:

receiving a cost structure for constructing a classification model using a data set;

instructing a plurality of human reviewing resources to search through the data set and select one or more instances of a class that satisfy at least one criterion, wherein the plurality of human reviewing resources are provided with a definition of the class;

training the classification model with the one or more instances from the plurality of human reviewing resources;

determining when an expected gain for performing additional searches by the plurality of human reviewing resources as a function of the cost structure is lower than a given threshold; and

in response to determining that the expected gain as a function of the cost structure is lower than the given threshold, instructing the plurality of human reviewing resources that was searching through the data set to label one or more examples from the data set.

2. The method of claim 1, wherein the plurality of human reviewing resources include resources for labeling the one or more instances and resources for searching for the one or more instances.

3. The method of claim 2, further comprising:

receiving a budget for constructing the classification model; and

allocating a portion of the budget to the plurality of human reviewing resources for searching the one or more instances and a remaining portion of the budget to the plurality of human reviewing resources for labeling the one or more instances.

4. The method of claim 1, wherein the one or more instances are minority class instances.

5. The method of claim 1, wherein the plurality of human reviewing resources are instructed to label the one or more examples by at least one of:

uncertainty sampling and boosted disagreement with query-by-committee.

6. The method of claim 1, further comprising allowing the plurality of human reviewing resources to query example web pages.

7. The method of claim 1, further comprising allowing the plurality of human reviewing resources to instruct an oracle for search engine queries that tend to reveal the class of interest.

8. The method of claim 1, further comprising allowing the plurality of human reviewing resources to query online resources that contain pointers to the class of interest.

9. The method of claim 1, further comprising instructing the plurality of human reviewing resources to discontinue labeling the one or more examples from the data set in response to determining that an expected gain for performing additional labeling by the plurality of human reviewing resources as a function of the cost structure is lower than a second threshold.

10. A system for data acquisition for construction of classification models that incorporates multiple human reviewing resources, the system comprising:

a processor that;

receives a cost structure for constructing a classification model using a data set;

instructs a plurality of human reviewing resources to search through the data set and select one or more instances of a class that satisfy at least one criterion, wherein the plurality of human reviewing resources are provided with a definition of the class;

trains the classification model with the one or more instances from the plurality of human reviewing resources;

determines when an expected gain for performing additional searches by the plurality of human reviewing resources as a function of the cost structure is lower than a given threshold; and

in response to determining that the expected gain as a function of the cost structure is lower than the given threshold, instructs the plurality of human reviewing resources that was searching through the data set to label the data set.

11. The system of claim 10, wherein the plurality of human reviewing resources include resources for labeling the one or more instances and resources for searching for the one or more instances.

12. The system of claim 11, wherein the processor is further configured to:

receive a budget for constructing the classification model; and

allocate a portion of the budget to the plurality of human reviewing resources for searching the one or more instances and a remaining portion of the budget to the plurality of human reviewing resources for labeling the one or more instances.

13. The system of claim 10, wherein the one or more instances are minority class instances.

14. The system of claim 10, wherein the plurality of human reviewing resources are instructed to label the one or more examples by at least one of:

uncertainty sampling and boosted disagreement with query-by-committee.

15. The system of claim 10, wherein the processor is further configured to allow the plurality of human reviewing resources to query example web pages.

16. The system of claim 10, wherein the processor is further configured to allow the plurality of human reviewing resources to instruct an oracle for search engine queries that tend to reveal the class of interest.

17. The system of claim 10, wherein the processor is further configured to allow the plurality of human reviewing resources to query online resources that contain pointers to the class of interest.

18. The system of claim 10, wherein the processor is further configured to instruct the plurality of human reviewing resources to discontinue labeling the one or more examples from the data set in response to determining that an expected gain for performing additional labeling by the plurality of human reviewing resources as a function of the cost structure is lower than a second threshold.

19. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for data acquisition for construction of classification models that incorporates multiple human reviewing resources, the method comprising:

receiving a cost structure for constructing a classification model using a data set;

instructing a plurality of human reviewing resources to search through the data set and select one or more instances of a class that satisfy at least one criterion, wherein the plurality of human reviewing resources are provided with a definition of the class;

training the classification model with the one or more instances from the plurality of human reviewing resources;

determining when an expected gain for performing additional searches by the plurality of human reviewing resources as a function of the cost structure is lower than a given threshold; and

in response to determining that the expected gain as a function of the cost structure is lower than the given threshold, instructing the plurality of human reviewing resources that was searching through the data set to label one or more examples from the data set.