Systems and methods for conducting internet content usage experiments

• US 7,747,681 B2
• Filed: 01/03/2008
• Issued: 06/29/2010
• Est. Priority Date: 12/20/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kromann, Paul K., Reeve, Thomas S., Cohen, Jules S.
• Primary Examiner(s): Flynn; Nathan
• Assistant Examiner(s): Siddiqi; Mohammad A
• Application Number:
  US11/968,935
• Publication Number:
  US 20080104167A1
• Time in Patent Office: 908 Days
• Field of Search: 709203-205, 709217-219, 709223-226, 709/229, 717/168, 717/174, 715/744, 715/745
• US Class Current: 709/203
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A system and method for efficiently executing a plurality of live parallel Internet content usage experiments using a large pool of randomly selected participating users (separated into control and test groups) while minimally disturbing offered content is provided. In an illustrative implementation, a content provider communicates the parameters for a content usage experiment to a content server. A complimentary test subject verification script operates on the content server that determines which client computing devices are to be included in the experiment group. In operation, a participating client computing device requests content from the content provider'"'"'s server. The content provider'"'"'s server executes the test subject verification script to determine if the client computing device is to be considered as a test subject. The content usage of the experiment content and non-experiment content is tracked and compared to each other to determine if the experiment content impacted content usage.

Claims

1. A computer-implemented method of executing a plurality of live content usage experiments, comprising:

providing at a content server a unique identification tag to a content requesting client computing device;

processing at the content server the unique identification tag upon receiving a request for content from the client computing device at the content server to verify if the client computing device is a test subject;

offering at the content server non-experiment content to the client computing device;

offering at the content server experiment content to the client computing device only if the client computing device is verified as a test subject;

tracking content usage at the content server of experiment content and non-experiment content offered to the client computing device to produce content usage data; and

analyzing the content usage data to compare usage of experiment content versus non-experiment content to determine if experiment content was used more often than non-experiment content.

2. The computer-implemented method recited in claim 1, wherein providing a unique identification tag further comprises determining if the requesting client computing device has a unique identification tag and assigning a new unique identification tag to the client computing device if the client computing device does not have a unique identification tag.

3. The computer-implemented method recited in claim 2, wherein determining if the requesting client computing device has a unique identification tag comprises processing COOKIES that are part of an HTTP header communicated as part of a request for content by the client computing device.

4. The computer-implemented method recited in claim 1, wherein processing the unique identification tag to verify if said client computing device is a test subject comprises performing a hashing function on the received unique identification tag to produce an integer and comparing the integer with a predefined range to verify if the requesting client computing device is a test subject.

5. The computer-implemented method recited in claim 1, further comprising incorporating the analyzed experiment content into non-experiment content if the data indicates the analyzed experiment content was more often used.

6. The computer-implemented method recited in claim 1, wherein providing, processing, and offering are performed by a content application operating on the content server communicating with the client computing devices, the content application cooperating with at least one configuration file having pre-defined content usage experiment parameters and at least one server side object that performs the verification of test subjects.

7. A computer-implemented method of executing live content usage experiments, comprising:

receiving at a content server a request to access live content from a client computing device;

processing the request at the content server to identify a unique identification tag associated with the client computing device;

processing at the content server the unique identification tag to verify if the client computing device is a test subject;

offering experiment content to the client computing device if the client computing device is verified as a test subject;

tracking content usage at the content server of experiment content and non-experiment content to produce content usage data; and

analyzing the content usage data to compare usage of experiment content versus non-experiment content to determine the efficacy of the experiment content.

8. The computer-implemented method of claim 7, wherein processing the request to identify a unique identification tag associated with the client computing device comprises processing the request to identify a GUID associated with the client computing device.

9. The computer-implemented method of claim 7, wherein processing at the content server said unique identification tag to verify if said client computing device is a test subject comprises associating a unique identification tag with the client computing device if it is determined the client computing device does not have a unique identifier associated with it.

10. The computer-implemented method of claim 7, wherein offering experiment content to the client computing device if the client computing device is verified as a test subject comprises executing a hashing function on the unique identification tag and determining whether a result of the hashing function falls within a range of values.

11. The computer-implemented method of claim 10, wherein executing a hashing function on the unique identification tag comprises executing a hashing function on a GUID.

12. The computer-implemented method of claim 7, wherein analyzing said content usage data to determine the efficacy of the experiment content comprises determining whether experiment content was accessed more than non-experiment content.

13. A system for performing live content usage experiments, comprising:

a processor adapted to execute computer-readable instructions;

memory communicatively coupled to said processor, said memory having stored therein computer-readable instructions that, if executed by the processor, cause the processor to perform operations comprising;

receiving requests for content from a plurality of client computing devices;

identifying a unique identifier for each of the plurality of client computing devices;

processing the unique identifiers associated with the client computing devices to identify client computing devices associated with a test group and to identify client computing devices associated with a control group;

offering experiment content to client computing devices identified as associated with the test group;

tracking content usage at said content server of experiment content and non-experiment content to produce content usage data; and

analyzing said content usage data to compare usage of experiment content versus non-experiment content.

14. The system of claim 13, wherein identifying a unique identifier for each of the plurality of client computing devices comprises identifying a GUID associated with a client computing device.

15. The system of claim 13, wherein identifying a unique identifier for each of the plurality of client computing devices comprises associating a unique identifier with a client computing device if it is determined the client computing device does not have a unique identifier associated with it.

16. The system of claim 13, wherein processing the unique identifiers associated with the client computing devices to identify client computing devices associated with a test group and to identify client computing devices associated with a control group comprises executing a hashing function on unique identifiers and determining whether results of the hashing function fall within a range of values.

17. The system of claim 16, wherein client computing devices are identified as associated with a test group if results of the hashing function fall within a range of values, and client computing devices are identified as associated with a control group if results of the hashing function fall outside a range of values.

18. The system of claim 16, wherein executing a hashing function on unique identifiers comprises executing a hashing function on GUIDs associated with client computing devices.

19. The system of claim 13, wherein analyzing said content usage data to compare usage of experiment content versus non-experiment content comprises analyzing said content usage data to determine the efficacy of the experiment content.

20. The system of claim 13, said memory further comprising instructions for performing the following:

determining whether the request for content comprises a request for content that is part of an experiment; and

determining whether an experiment is active.