Content item relevance based on presentation data

• US 9,053,129 B1
• Filed: 03/14/2013
• Issued: 06/09/2015
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Covell, Michele, Baluja, Shumeet
• Primary Examiner(s): Pham, Hung Q
• Application Number:
  US13/803,180
• Time in Patent Office: 817 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06Q 30/0241 Advertisements

Abstract

Methods, systems, and apparatus for determining content item quality based on content item presentation data are presented. In one aspect, a method includes storing a first count of occurrences of a pair of content items being presented; storing a second count of second occurrences of a first content item of the pair being interacted with when it was presented in a lesser position in a resource relative to a second content item of the pair when the second content item of the pair was not interacted with; determining that the first count meets a first threshold; determining that a ratio of the second count to the first count meets a second threshold; and storing correlation data for the second content item based on the ratio.

Claims

1. A method comprising:

determining, by a computer device, a first collection of historical occurrences of one or more content item pairs being presented together in association with a single resource;

storing, by the computer device, in a first data structure an entry that indicates a first count of the first collection of historical occurrences of the one or more content item pairs;

determining, by the computer device and for each content item pair of the one or more content item pairs, a second collection of historical occurrences of a first content item of the content item pair being interacted with when it was presented in a lesser position in a block of content items, wherein the lesser position is not a first position in the block of content items;

for the content item pair including the first content item of the content item pair and a second content item of the content item pair, storing, by the computer device, in a second data structure a second count of a number of times in the second collection of historical occurrences that the first content item was interacted with while in a less prominent position relative to the second content item of the content item pair while the second content item was not interacted with;

determining, by the computer device, entries in the first data structure where the first count exceeds a first threshold;

for each determined entry and by the computer device, locating a corresponding entry in the second data structure and determining if a ratio of the second count for the corresponding entry in the second data structure to the first count for the entry in the first data structure exceeds a predetermined second threshold; and

when the ratio exceeds the predetermined second threshold, determining, by the computer device, that a dichotomy exists and storing information about the determined dichotomy in a data storage system.

2. The method of claim 1, wherein the one or more content item pairs are presented in response to a single request for content for presentation with the single resource.

3. The method of claim 1, wherein the one or more content item pairs are presented in a same ad block of the single resource.

4. The method of claim 1, wherein occurrences of the first content item being interacted with include occurrences of the first content item being clicked on.

5. The method of claim 1, wherein determining the first collection of historical occurrences includes evaluating a database that includes impression data for ads presented responsive to ad requests.

6. The method of claim 1, wherein determining the second collection of historical occurrences includes evaluating a database that includes click-through data after presentation of ads in an ad block.

7. The method of claim 1, further comprising using the information of the determined dichotomy to select content to deliver to an entity.

8. The method of claim 1, further comprising:

receiving a request for content;

identifying a plurality of candidate content items responsive to the request;

determining a score for each candidate content item;

adjusting scores for the plurality of candidate content items based on stored dichotomy information;

selecting a content item from the plurality of candidate content items based at least in part on the adjusted scores; and

providing the selected content item responsive to the received request.

9. The method of claim 8, wherein determining a score for each candidate content item includes determining whether pairs of the plurality of candidate content items define a dichotomy based on the stored dichotomy information, and adjusting a score of at least one of the plurality of candidate content items based on a relative strength of the defined dichotomy.

10. The method of claim 9 wherein the relative strength is determined based on a ratio of second counts to first counts for a given pair.

11. A method performed by one or more data processing apparatus, the method comprising:

determining a first collection of historical occurrences of a pair of content items being presented together in association with one or more resources;

storing, in a first data structure, a first entry that indicates a first count of the first collection of historical occurrences;

determining a second collection of historical occurrences of a first content item of the pair of content items being interacted with when it was presented in a lesser position in a resource relative to a second content item of the pair of content items when the second content item was not interacted with;

storing, in a second data structure, a second entry that indicates a second count of the second collection of historical occurrences;

determining that the first entry in the first data structure meets a first threshold;

in response to the determination that the first entry in the first data structure meets the first threshold, determining that a ratio of the second count for the second entry in the second data structure to the first count for the first entry in the first data structure meets a second threshold; and

in response to the determination that the ratio of the second count for the second entry in the second data structure to the first count for the first entry in the first data structure meets the second threshold, storing correlation data for the second content item in a data storage system, the correlation data being based on the ratio of the second count for the second entry in the second data structure to the first count for the first entry in the first data structure meets the second threshold.

12. A system comprising:

a data processing apparatus; and

a data store storing instructions that, when executed by the data processing apparatus, cause the data processing apparatus to perform operations comprising;

determining a first collection of historical occurrences of one or more content item pairs being presented together in association with a single resource;

storing in a first data structure an entry that indicates a first count of the first collection of historical occurrences of the one or more content item pairs;

determining, for each content item pair of the one or more content item pairs, a second collection of historical occurrences of a first content item of the content item pair being interacted with when it was presented in a lesser position in a block of content items, wherein the lesser position is not a first position in the block of content items;

for the content item pair including the first content item of the given content item pair and a second content item of the content item pair, storing in a second data structure a second count of a number of times in the second collection of historical occurrences that the first content item was interacted with while in a less prominent position relative to the second content item of the content item pair while the second content item was not interacted with;

determining entries in the first data structure where the first count exceeds a first threshold;

for each determined entry, locating a corresponding entry in the second data structure and determining if a ratio of the second count for the corresponding entry in the second data structure to the first count for the entry in the first data structure exceeds a predetermined second threshold; and

when the ratio exceeds the predetermined second threshold, determining that a dichotomy exists and storing information about the determined dichotomy in a data storage system.

13. The system of claim 12, wherein the one or more content item are presented in response to a single request for content for presentation with the single resource.

14. The system of claim 12, wherein determining the first collection of historical occurrences includes evaluating a database that includes impression data for ads presented responsive to ad requests.

15. The system of claim 12, wherein determining the second collection of historical occurrences includes evaluating a database that includes click-through data after presentation of ads in an ad block.

16. The system of claim 12, wherein the operations further comprise using the information of the determined dichotomy to select content to deliver to an entity.

17. The system of claim 12, wherein the operations further comprise:

receiving a request for content;

identifying a plurality of candidate content items responsive to the request;

determining a score for each candidate content item;

adjusting scores for the plurality of candidate content items based on stored dichotomy information;

selecting a content item from the plurality of candidate content items based at least in part on the adjusted scores; and

providing the selected content item responsive to the received request.

18. The system of claim 17, wherein determining a score for each candidate includes determining whether pairs of the plurality of candidate content items define a dichotomy based on the stored dichotomy information, and adjusting a score of at least one of the plurality of candidate content items based on a relative strength of the defined dichotomy.

19. The system of claim 18 wherein the relative strength is determined based on a ratio of second counts to first counts for a given pair.

20. A computer readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising:

determining a first collection of historical occurrences of one or more content item pairs being presented together in association with a single resource;

storing in a first data structure an entry that indicates a first count of the first collection of historical occurrences of the one or more content item pairs;

determining, for each content item pair of the one or more content item pairs, a second collection of historical occurrences of a first content item of the content item pair being interacted with when it was presented in a lesser position in a block of content items, wherein the lesser position is not a first position in the block of content items;

for the content item pair including the first content item of the content item pair and a second content item of the content item pair, storing in a second data structure a second count of a number of times in the second collection of historical occurrences that the first content item was interacted with while in a less prominent position relative to the second content item of the content item pair while the second content item was not interacted with;

determining entries in the first data structure where the first count exceeds a first threshold;

for each determined entry, locating corresponding entries in the second data structure and determining if a ratio of the second count for the corresponding entry in the second data structure to the first count for the entry in the first data structure exceeds a predetermined second threshold; and

when the ratio exceeds the predetermined second threshold, determining that a dichotomy exists and storing information about the determined dichotomy in a data storage system.