Adjust similar users identification based on performance feedback

• US 8,874,589 B1
• Filed: 07/16/2012
• Issued: 10/28/2014
• Est. Priority Date: 07/16/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Liu, Jia, Bai, Yijian, Patil, Manojav, Ravichandran, Deepak, Jiampojamarn, Sittichai, Ponnekanti, Shankar
• Primary Examiner(s): Badawi, Sherief
• Assistant Examiner(s): HARMON, COURTNEY N
• Application Number:
  US13/550,073
• Time in Patent Office: 834 Days
• Field of Search: 707/749
• US Class Current: 707/749
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A method of setting a threshold similarity score value for a first plurality of network user identifiers. The first plurality of network user identifiers, a second plurality of network user identifiers and characteristic data associated with the network user identifiers is received. A performance target and an experimental threshold similarity score value are designated. A similarity score between the first and second plurality of network user identifiers is calculated. Performance statistics data for each of the second plurality of network user identifiers having a similarity score greater than or equal to the experimental threshold similarity score value is collected and compared to the similarity score of the network user identifier. Based on the comparison, the experimental threshold similarity score value is adjusted to a similarity score value that achieves the performance target and the threshold similarity score value is set to the adjusted experimental threshold similarity score value.

Claims

1. A computerized method of setting a threshold similarity score value for a first plurality of network device identifiers, the method comprising:

receiving, at a processing circuit, a first plurality of network device identifiers and characteristic data associated with network activity of each of the first plurality of network device identifiers;

receiving, at the processing circuit, a second plurality of network device identifiers that do not appear in the first plurality of network device identifiers and characteristic data associated with network activity of each of the second plurality of network device identifiers;

calculating, for each network device of the second plurality of network device identifiers, a similarity score that represents a degree of similarity between the characteristic data for the network device identifier of the second plurality and the characteristic data for the network device identifiers of the first plurality;

designating, by the processing circuit, a performance target relating to a factor indicative of an interest in or usefulness of content placed on a webpage, the performance target used to identify a smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

designating, by the processing circuit, a threshold similarity score value selected as a starting value for determining a lowest similarity score value that is used to identify the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

identifying a first number of network device identifiers from the second plurality that have similarity scores above the threshold similarity score value;

receiving, by the processing circuit, for each of the identified network device identifiers of the second plurality that have a similarity score above the threshold similarity score value, performance statistics data corresponding to the factor related to the designated performance target;

aggregating the performance statistics data of each of the identified network device identifiers to determine an aggregate performance statistics data;

determining, from the aggregate performance statistics data, that the first number of network device identifiers is not the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

iteratively adjusting, responsive to determining that the first number of network device identifiers is not the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target, the threshold similarity score value to identify the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target; and

setting the adjusted threshold similarity score value to an experimental threshold similarity score value that represents the lowest similarity score value that identifies the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target.

2. The method of claim 1, wherein the designated performance target is at least one of a number of devices that qualify as a similar device to the first plurality of network device identifiers, a click through rate for content, a number of conversions, a conversion per dollar and a total revenue.

3. The method of claim 1, wherein receiving performance statistics data comprises receiving performance statistics data for a randomly selected subset of the second plurality of network device identifiers having a similarity score greater than or equal to the threshold similarity score value.

4. The method of claim 1, wherein the comparing further comprises plotting the performance statistics data versus the similarity score for each of the second plurality of network device identifiers having a similarity score greater than or equal to the threshold similarity score value.

5. The method of claim 3, further comprising applying a smoothing function to the performance statistics data after the plotting step.

6. The method of claim 3, further comprising interpolating the performance statistics data responsive to determining a distribution of the performance statistics data does not cover the designated performance target.

7. The method of claim 1, wherein the adjusting comprises selecting the lowest similarity score that identifies the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the performance target as the experimental threshold value.

8. The method of claim 1, wherein the adjusting comprises selecting a lowest similarity score that identifies the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the performance target with a predetermined level of confidence as the experimental threshold value.

9. The method of claim 1, further comprising repeating the method using the adjusted threshold similarity score value as the threshold similarity score value.

10. A system for setting a threshold similarity score value for a first plurality of network device identifiers, comprising:

a hardware processing circuit operable to;

receive the first plurality of network device identifiers and characteristic data associated with network activity of each of the first plurality of network device identifiers;

receive a second plurality of network device identifiers that do not appear in the first plurality of network device identifiers and characteristic data associated with network activity of each of the second plurality of network device identifiers;

calculate, for each network device of the second plurality of network device identifiers, a similarity score that represents a degree of similarity between the characteristic data for the network device identifier of the second plurality and the characteristic data for the network device identifiers of the first plurality;

designate a performance target relating to a factor indicative of an interest in or usefulness of content placed on a webpage, the performance target used to identify a smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

designate a threshold similarity score value selected as a starting value for determining a lowest similarity score value that is used to identify the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

identify a first number of network device identifiers from the second plurality that have similarity scores above the threshold similarity score value;

receive, for each of the identified network device identifiers of the second plurality that have a similarity score above the threshold similarity score value, performance statistics data corresponding to the factor related to the designated performance target;

aggregate the performance statistics data of each of the identified network device identifiers to determine an aggregate performance statistics data;

determine, from the aggregate performance statistics data, that the first number of network device identifiers is not the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

iteratively adjust, responsive to determining that the first number of network device identifiers is not the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target, the threshold similarity score value to identify the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target; and

set the adjusted threshold similarity score value to an experimental threshold similarity score value that represents the lowest similarity score value that identifies the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target.

11. The system of claim 10, wherein the designated performance target is at least one of a number of devices that qualify as a similar device to the first plurality of network device identifiers, a click through rate for content, a number of conversions, a conversion per dollar and a total revenue.

12. The system of claim 10, wherein in receiving the performance statistics data, the processing circuit is further operable to receive performance statistics data for a randomly selected subset of the second plurality of network device identifiers having a similarity score greater than or equal to the threshold similarity score value.

13. The system of claim 10, wherein in the comparing, the processing circuit is further operable to plot the performance statistics data versus the similarity score for each of the second plurality of network device identifiers having a similarity score greater than or equal to the threshold similarity score value.

14. The system of claim 13, wherein after the plotting, the processing circuit is further operable to apply a smoothing function to the performance statistics data.

15. The system of claim 13, wherein the processing circuit is further operable to interpolate the performance statistics data responsive to determining a distribution of the performance statistics data does not cover the designated performance target.

16. The system of claim 10, wherein in the adjusting, the processing circuit is further operable to select the lowest similarity score that identifies the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the performance target as the experimental threshold value.

17. The system of claim 10, wherein in the adjusting, the processing circuit is further operable to select a lowest similarity score that identifies the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the performance target with a predetermined level of confidence as the experimental threshold similarity score value.

18. The system of claim 10, wherein the processing circuit is further operable to repeat the calculating, the identifying, the receiving, the iteratively adjusting and the setting using the adjusted threshold similarity score value as the threshold similarity score value.

19. A computer-readable storage device having instructions thereon that cause one or more processors to perform operations, the operations comprising:

receiving a first plurality of network device identifiers and characteristic data associated with network activity of each of the first plurality of network device identifiers;

receiving a second plurality of network device identifiers that do not appear in the first plurality of network device identifiers and characteristic data associated with network activity of each of the second plurality of network device identifiers;

calculating, for each network device of the second plurality of network device identifiers, a similarity score that represents a degree of similarity between the characteristic data for the network device identifier of the second plurality and the characteristic data for the network device identifiers of the first plurality;

designating a performance target relating to a factor indicative of an interest in or usefulness of content placed on a webpage, the performance target used to identify a smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

designating a threshold similarity score value selected as a starting value for determining a lowest similarity score value that is used to identify the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

identifying a first number of network device identifiers from the second plurality that have similarity scores above the threshold similarity score value;

receiving, for each of the identified network device identifiers of the second plurality that have a similarity score above the threshold similarity score value, performance statistics data corresponding to the factor related to the designated performance target;

aggregating the performance statistics data of each of the identified network device identifiers to determine an aggregate performance statistics data;

determining, from the aggregate performance statistics data, that the first number of network device identifiers is not the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target;

iteratively adjusting, responsive to determining that the first number of network device identifiers is not the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target, the threshold similarity score value to identify the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target; and

setting the adjusted threshold similarity score value to an experimental threshold similarity score value that represents the lowest similarity score value that identifies the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the designated performance target.

20. The computer-readable storage device of claim 19, wherein the adjusting step comprises selecting a lowest similarity score that identifies the smallest number of network device identifiers of the second plurality of network device users that have performance statistics data relating to the factor, which when aggregated, achieves the performance target, as the experimental threshold value.