Systems and methods for conducting internet content usage experiments

• US 7,343,390 B2
• Filed: 12/20/2000
• Issued: 03/11/2008
• Est. Priority Date: 12/20/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kromann, Paul K., Reeve, Thomas S., Cohen, Jules S.
• Primary Examiner(s): Flynn; Nathan
• Assistant Examiner(s): Siddiqi; Mohammad
• Application Number:
  US09/741,362
• Publication Number:
  US 20020078141A1
• Time in Patent Office: 2,638 Days
• Field of Search: 709/246, 709/208, 709203-205, 709/206, 709217-219, 709/229, 709223-226, 717/168, 717/174, 715/744, 715/745
• US Class Current: 709/203
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A system and method for efficiently executing a plurality of live parallel Internet content usage experiments using a large pool of randomly selected participating users (separated into control and test groups) while minimally disturbing offered content is provided. In an illustrative implementation, a content provider communicates the parameters for a content usage experiment to a content server. A complimentary test subject verification script operates on the content server that determines which client computing devices are to be included in the experiment group. In operation, a participating client computing device requests content from the content provider'"'"'s server. The content provider'"'"'s server executes the test subject verification script to determine if the client computing device is to be considered as a test subject. The content usage of the experiment content and non-experiment content is tracked and compared to each other to determine if the experiment content impacted content usage.

Claims

1. A method of executing a plurality of live content usage experiments comprising the acts of:

providing a unique identification tag to a content requesting client computing device by a content server;

processing said unique identification tag upon receiving a request for content from said client computing device by said content server to verify if said client computing device is a test subject;

offering experiment content to said client computing device if said client computing device is verified as a test subject;

tracking content usage at said content server of experiment content and non-experiment content offered to said client computing device to produce content usage data; and

analyzing said content usage data to compare usage of experiment content versus non-experiment content to determine if experiment content was used more often than non-experiment content.

2. The method recited in claim 1, wherein said act of providing unique identification tag further comprises the acts of determining if said requesting client computing device has a unique identification tag and assigning a new unique identification tag to said client computing device if said client computing device does not have a unique identification tag.

3. The method recited in claim 2, wherein said determining act comprises processing COOKIES that are part of HTTP headers communicated as part of requests for content by said client computing devices.

4. The method recited in claim 1, wherein said processing step further comprises the acts of performing a hashing function on said received unique identification tags to produce an integer and comparing said integer with a predefined range to verify if said requesting client computing device is a test subject.

5. The method recited in claim 1, further comprising the act of incorporating said analyzed experiment content into non-experiment content if said data indicates said analyzed experiment content was more often used.

6. The method recited in claim 1, wherein said providing, processing, and offering acts are performed by a content application operating on said content server communicating with said client computing devices, said content application cooperating with at least one configuration file having pre-defined content usage experiment parameters and at least one server side object that performs the verification of test subjects.

7. The method recited in claim 6, wherein said communicating act is realized through a communications network comprising any of a fixed-wire LAN, a wireless LAN, a fixed-wire WAN, a wireless WAN, a fixed-wire intranet, a wireless intranet, and the Internet.

8. A computer-readable medium bearing computer-readable instructions for instructing a computer to carry out the acts recited in claim 1.

9. A method to execute live content usage experiments comprising the acts of:

requesting content by at least one client computing device, said at least one client computing device cooperating with a content server over a communications network;

receiving content by said at least one client computing device from said content server, said content being experiment content if said at least one client computing device has a unique identification tag and is verified by said content server using the unique identification tag as a test subject;

tracking content usage for said received content by said at least one computing device; and

communicating said content usage information by said at least one client computing device to said content server, wherein said content usage information is analyzed by said content server to determine the efficacy of experiment content.

10. The method recited in claim 9, wherein said requesting act comprises sending a request by said at least one client computing device to said content server, said request comprising a header having data indicative of said client computing device.

11. The method recited in claim 10, wherein said sending act comprises attaching COOKIES to said request, said COOKIES comprising a unique identification tag that may be used by said content server to verify if said at least one client computing device is a test subject.

12. The method recited in claim 9, wherein said requesting and receiving acts are performed by a computing application operating on said at least one client computing device, said computing application having an interface to procure requests for content and to display received content.

13. A computer-readable medium bearing computer-readable instructions for instructing a computer to carry out the acts recited in claim 9.

14. A system for executing a plurality of live content usage experiments comprising:

a content usage experiment system, said content usage experiment system comprising experiment configuration data having at least one experiment parameter, a content server that performs test subject verification processes using said at least one experiment parameter from said experiment configuration data, and content data stores; and

a communications network, to communicate said content by said content usage experiment system to cooperating client computing devices,

wherein said content usage experiment system receives a request for content from a client computing device, processes said request for content to determine if said client computing device has an assigned unique identification tag, and executes an application to perform said test subject verification processes using said unique identification tag to determine if said client computing device is to be a test subject,

wherein said content usage experiment system tracks usage of experiment content, and analyzes usage of the experiment content to determine the efficacy of the experiment content.

15. The system recited in claim 14, wherein said computing application comprises a server side object that receives said unique identification tag of said client computing device as input and performs a hashing function to produce an integer, said server side object comparing said integer to a pre-defined range to determine if said client computing device is to be a test subject.

16. The system recited in claim 15, wherein said server side object uses a hash divisor to perform said hashing function.

17. The system recited in claim 15, wherein said experiment content is communicated by said content usage experiment system to said client computing device that is determined to be said test subject.

18. The system recited in claim 17, wherein said experiment content is only communicated by said content usage experiment system to said client computing device if the experiment to which the experiment content is associated has not expired.

19. The system recited in claim 14, wherein said communications network comprises any of a fixed-wire LAN, a wireless LAN, a fixed-wire WAN, a wireless WAN, a fixed-wire intranet, a wireless intranet, and the Internet.

20. The system recited in claim 14, wherein said content stores comprise an experiment content store, a non-experiment content store, and a content usage data store.

21. In a computer network environment a content usage experiment system comprising:

a content server, said content server in communication with a client computing device over a communications network, said content server receiving requests for content from said client computing device, said client computing device having either been previously assigned a unique identification tag or requiring a unique identification tag; and

a computing application executing on said content server, said computing application comprising a server object that accepts as input said unique identification tag and processes said unique identification tag with a hashing algorithm to produce an integer, said server side object comparing the calculated integer with a range for pre-defined test subjects to determine if said client computing device is to be a test subject, said computing application communicating experiment content from a cooperating experiment content data store to said client computing device if said client computing device is determined to be a test subject over said communications network and communicating non-experiment content from a non-experiment data store to said client computing device if determined not be a test subject over said communications network,

wherein content usage data is tracked and stored by said content application for said communicated experiment content and non-experiment content in a cooperating content usage data store, said computing application performing an analysis of said content usage data to determine the efficacy of said communicated experiment content as compared to said non-experiment content.

22. The system recited in claim 21, wherein said analysis employs usage variables comprising the number of time specific experiment and/or non-experiment content was interacted with by client computing devices, the number of unique client computing devices interacting with specific experiment and/or non-experiment content, the time/date of interaction by client computing devices, and the pattern of interaction with experiment and/or non-experiment specific content.