Systems and methods for conducting internet content usage experiments

• US 8,255,458 B2
• Filed: 05/21/2010
• Issued: 08/28/2012
• Est. Priority Date: 12/20/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cohen, Jules S., Kromann, Paul K., Reeve, Thomas S.
• Primary Examiner(s): Avellino, Joseph
• Assistant Examiner(s): McLeod, Marshall
• Application Number:
  US12/784,722
• Publication Number:
  US 20100228822A1
• Time in Patent Office: 830 Days
• Field of Search: 709/203, 709/204, 709/205, 709/206, 709/224, 709/227
• US Class Current: 709/203
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A system and method for efficiently executing a plurality of live parallel Internet content usage experiments using a large pool of randomly selected participating users (separated into control and test groups) while minimally disturbing offered content is provided. In an illustrative implementation, a content provider communicates the parameters for a content usage experiment to a content server. A complimentary test subject verification script operates on the content server that determines which client computing devices are to be included in the experiment group. In operation, a participating client computing device requests content from the content provider'"'"'s server. The content provider'"'"'s server executes the test subject verification script to determine if the client computing device is to be considered as a test subject. The content usage of the experiment content and non-experiment content is tracked and compared to each other to determine if the experiment content impacted content usage.

Claims

1. A computer-implemented method for serving data, comprising:

on a computing system, receiving requests for a content item from a plurality of computing devices;

on a computing system, identifying for each of the plurality of computing devices identifying data;

on a computing system, determining for each of the plurality of computing devices using the identifying data whether to transmit experimental data or non-experimental data;

on a computing system, transmitting experimental data to a first set of the plurality of computing devices;

on a computing system, transmitting non-experimental data to a second set of the plurality of computing devices; and

on a computing system, tracking usage of the experimental data by the first set of plurality of computing devices and tracking usage of the non-experimental data by the second set of plurality of computing devices to create content usage data.

2. The computer-implemented method recited in claim 1, wherein identifying for each of the plurality of computing devices identifying data comprises determining if a computing device has a unique identifier and assigning a new unique identifier to the computing device if the computing device does not have a unique identifier.

3. The computer-implemented method recited in claim 2, wherein determining if a computing device has a unique identifier comprises processing COOKIES that are part of an HTTP header communicated as part of a request for content by the computing device.

4. The computer-implemented method recited in claim 1, wherein determining for each of the plurality of computing devices using the identifying data whether to transmit experimental data or non-experimental data comprises performing a hashing function on the identifying data to produce a value and comparing the value with a predefined range to determine whether the computing device should receive experimental data or non-experimental data.

5. The computer-implemented method recited in claim 1, further comprising processing the content usage data to determine whether the experimental data impacted usage relative to the non-experimental data.

6. The computer-implemented method recited in claim 5, wherein processing the content usage data to determine whether the experimental data impacted usage relative to the non-experimental data comprises processing the content usage data to determine the number of times data was interacted with by client computing devices.

7. The computer-implemented method recited in claim 5, further comprising incorporating the experimental data into the non-experimental data upon determining the experimental data increased frequency of use.

8. The computer-implemented method of claim 1, wherein determining for each of the plurality of computing devices using the identifying data whether to transmit experimental data or non-experimental data comprises determining whether the content item is the subject of an experiment.

9. The computer-implemented method of claim 8, wherein determining for each of the plurality of computing devices using the identifying data whether to transmit experimental data or non-experimental data further comprises determining whether the experiment is active.

10. A system for serving data, comprising:

a processor adapted to execute computer-readable instructions;

computing memory communicatively coupled to said processor, said memory having stored therein computer-readable instructions that, if executed by the processor, cause the processor perform operations comprising;

receiving requests for a content item from a plurality of computing devices;

identifying for each of the plurality of computing devices identifying data;

determining for each of the plurality of computing devices using the identifying data whether to transmit experimental data or non-experimental data;

transmitting experimental data to a first set of the plurality of computing devices;

transmitting non-experimental data to a second set of the plurality of computing devices; and

tracking usage of the experimental data by the first set of plurality of computing devices and tracking usage of the non-experimental data by the second set of plurality of computing devices to create content usage data.

11. The system recited in claim 10, wherein identifying for each of the plurality of computing devices identifying data comprises identifying a GUID associated with a computing device.

12. The system recited in claim 10, wherein determining for each of the plurality of computing devices using the identifying data whether to transmit experimental data or non-experimental data comprises performing a hashing function on the identifying data to produce a value and comparing the value with a predefined range to determine whether the computing device should receive experimental data or non-experimental data.

13. The system recited in claim 12, wherein performing a hashing function comprises performing a hashing function on a GUID.

14. The system recited in claim 10, said computing memory further comprising instructions for performing the following:

processing the content usage data to determine whether the experimental data impacted usage relative to the non-experimental data.

15. The system recited in claim 14, wherein processing the content usage data to determine whether the experimental data impacted usage relative to the non-experimental data comprises processing the content usage data to determine the number of times data was interacted with by client computing devices.

16. The system recited in claim 14, said computing memory further comprising instructions for incorporating the experimental data into the non-experimental data upon determining the experimental data increased frequency of use.

17. A computer readable medium device comprising instructions that when executed by a computing processor cause the computing processor to perform operations comprising:

receiving requests for a content item from a plurality of computing devices;

identifying for each of the plurality of computing devices identifying data;

determining for each of the plurality of computing devices using the identifying data whether to transmit a first version of the content item or second version of the content item;

transmitting the first version of the content item to a first set of the plurality of computing devices;

transmitting the second version of the content item to a second set of the plurality of computing devices; and

tracking usage of the first version of the content item by the first set of plurality of computing devices and tracking usage of the second version of the content item by the second set of plurality of computing devices to create content usage data.

18. The computer readable medium device as recited in claim 17, wherein identifying for each of the plurality of computing devices identifying data comprises identifying a GUID associated with a computing device.

19. The computer readable medium device as recited in claim 17, further comprising instructions that when executed by a computing processor cause the computing processor to perform operations comprising:

processing the content usage data to determine whether the experimental data impacted usage relative to the non-experimental data.

20. The computer readable medium device as recited in claim 19, further comprising instructions that when executed by a computing processor cause the computing processor to perform operations comprising:

incorporating the experimental data into the non-experimental data upon determining the experimental data increased frequency of use.