Automated resource selection process evaluation
First Claim
1. A system comprising:
- one or more computers configured to perform operations comprising;
receiving a plurality of test queries;
generating, for each test query, a first group of resources corresponding to a first automated resource selection process and a second group of resources corresponding to a second automated resource selection process, the generating comprising, for each test query;
identifying a plurality of resources responsive to the test query,determining, for each resource in the plurality of resources, whether the first automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the first group of resources, anddetermining, for each resource in the plurality of resources, whether the second automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the second group of resources,wherein the determination whether the first automated resource selection process would classify a particular resource as to-be-indexed or not-to-be-indexed is made independently of the determination whether the second automated resource selection process would classify that particular resource as to-be-indexed or not-to-be-indexed.
2 Assignments
0 Petitions
Accused Products
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating resource selection processes. One method includes receiving test queries and generating a first group of resources corresponding to a first automated resource selection process and generating a second group of resources corresponding to a second automated resource selection process for each query. Another method includes generating a query results table for use in generating the groups of resources. The query results table maps queries to resources matched to the queries, and maps each resource to a respective score for the resource and the query, and one or more index selection signals for the resource.
-
Citations
31 Claims
-
1. A system comprising:
one or more computers configured to perform operations comprising; receiving a plurality of test queries; generating, for each test query, a first group of resources corresponding to a first automated resource selection process and a second group of resources corresponding to a second automated resource selection process, the generating comprising, for each test query; identifying a plurality of resources responsive to the test query, determining, for each resource in the plurality of resources, whether the first automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the first group of resources, and determining, for each resource in the plurality of resources, whether the second automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the second group of resources, wherein the determination whether the first automated resource selection process would classify a particular resource as to-be-indexed or not-to-be-indexed is made independently of the determination whether the second automated resource selection process would classify that particular resource as to-be-indexed or not-to-be-indexed. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
-
14. A computer-implemented method, comprising:
-
receiving a plurality of test queries; generating, for each test query, a first group of resources corresponding to a first automated resource selection process and a second group of resources corresponding to a second automated resource selection process, the generating comprising, for each test query; identifying a plurality of resources responsive to the test query, determining, for each resource in the plurality of resources, whether the first automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the first group of resources, and determining, for each resource in the plurality of resources, whether the second automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the second group of resources, wherein the determination whether the first automated resource selection process would classify a particular resource as to-be-indexed or not-to-be-indexed is made independently of the determination whether the second automated resource selection process would classify that particular resource as to-be-indexed or not-to-be-indexed. - View Dependent Claims (15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26)
-
-
27. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
-
receiving a plurality of test queries; generating, for each test query, a first group of resources corresponding to a first automated resource selection process and a second group of resources corresponding to a second automated resource selection process, the generating comprising, for each test query; identifying a plurality of resources responsive to the test query, determining, for each resource in the plurality of resources, whether the first automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the first group of resources, and determining, for each resource in the plurality of resources, whether the second automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the second group of resources, wherein the determination whether the first automated resource selection process would classify a particular resource as to-be-indexed or not-to-be-indexed is made independently of the determination whether the second automated resource selection process would classify that particular resource as to-be-indexed or not-to-be-indexed.
-
-
28. A system comprising:
one or more computers configured to perform operations comprising; receiving a plurality of test queries; selecting, for each test query, a first group of resources corresponding to a first automated resource selection process and a second group of resources corresponding to a second automated resource selection process, the selecting comprising, for each test query; identifying a plurality of resources responsive to the test query, determining, for each resource in the plurality of resources, whether the first automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the first group of resources, and determining, for each resource in the plurality of resources, whether the second automated resource selection process would classify the resource as to-be-indexed or not-to-be-indexed, and then selecting all resources classified as to-be-indexed as the second group of resources; obtaining a respective query-specific score for each resource in the first group of resources and each resource in the second group of resources; presenting for comparison by a user first search results for a first subset of the first group of resources and second search results for a second subset of the second group of resources, wherein the first search results are presented in an order derived from the respective query-specific scores for the resources in the first subset and the second search results are presented in an order derived from the respective query-specific scores for the resources in the second subset. - View Dependent Claims (29)
-
30. A method comprising:
-
receiving a plurality of test queries; selecting, for each test query, a first group of resources corresponding to a first automated resource selection process and a second group of resources corresponding to a second automated resource selection process, the selecting comprising, for each test query; identifying a plurality of resources responsive to the test query, determining, for each resource in the plurality of resources, whether the first automated resource selection process would classify the resource as to be indexed or not to be indexed, and then selecting all resources classified as to be indexed as the first group of resources, and determining, for each resource in the plurality of resources, whether the second automated resource selection process would classify the resource as to be indexed or not to be indexed, and then selecting all resources classified as to be indexed as the second group of resources; obtaining a respective query specific score for each resource in the first group of resources and each resource in the second group of resources; presenting for comparison by a user first search results for a first subset of the first group of resources and second search results for a second subset of the second group of resources, wherein the first search results are presented in an order derived from the respective query specific scores for the resources in the first subset and the second search results are presented in an order derived from the respective query specific scores for the resources in the second subset. - View Dependent Claims (31)
-
Specification