Systems and methods for determining a quality of provided items

• US 8,065,296 B1
• Filed: 09/29/2004
• Issued: 11/22/2011
• Est. Priority Date: 09/29/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Franz, Alexander Mark, Henzinger, Monika H.
• Primary Examiner(s): Yen; Syling
• Application Number:
  US10/952,501
• Time in Patent Office: 2,610 Days
• Field of Search: 707/1, 707/2, 707/4, 707/10, 707/100, 715/853, 715/523, 715/835, 715/857, 705/1, 709/206, 709/218, 713/165
• US Class Current: 707/723
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A system may provide items during a time period and determine a quality of the items provided during the time period using a time series model.

Claims

1. A method comprising:

providing items during a time period; and

determining an indication of quality of the items provided during the time period using a time series model; and

taking remedial measures when the indication of quality of the items is below an expectation, wherein the remedial measures comprise;

automatically removing a modification associated with the providing;

wherein the items include search results.

2. The method of claim 1 wherein the items include advertisements.

3. The method of claim 1 further comprising:

selecting a metric for the time series model, the metric being based on one or more user actions.

4. The method of claim 3 wherein the items comprise search results, and

wherein the metric comprises one or more of a percentage of times that a high ranking search result is selected, an average first click position, a percentage of searches that had long clicks, a percentage of searches that did not have another search within a first period of time, or a percentage of searches that did not have a reformulated search within a second period of time.

5. The method of claim 1 wherein the time period is configurable.

6. The method of claim 1 wherein the determining an indication of quality of items provided during the time period comprises:

predicting a range of metric values for the time period using the time series model.

7. The method of claim 6 wherein the determining an indication of quality of items provided during the time period further comprises:

determining an actual metric value for the time period, and

comparing the actual metric value to the predicted range of metric values.

8. The method of claim 7 further comprising:

taking remedial measures when the actual metric value for the time period falls outside the predicted range of metric values.

9. The method of claim 1 wherein the determining an indication of quality of the items provided during the time period comprises:

determining the indication of quality of the items by comparing a metric value for the time period to a predicted metric value determined using the time series model.

10. The method of claim 1 wherein the remedial measures further comprise:

sending a notification.

11. A method comprising:

modifying a system that provides a plurality of items;

recording user behavior relating to providing the plurality of items over a time period;

predicting user behavior for the time period, where predicting user behavior comprises determining a range of predicted values for the user behavior;

determining an indication of quality of the plurality of items provided during the time period based on the recorded user behavior and the predicted user behavior, where determining an indication of quality of the plurality of items comprises;

comparing a value representing the recorded user behavior to the determined range of predicted values, and

determining that the indication of quality is below an expectation when the value is outside the range of predicted values; and

taking remedial measures when the indication of quality is determined to be below the expectation, where the remedial measures comprise automatically removing the modification to the system.

12. The method of claim 11 wherein the predicting user behavior comprises:

predicting the user behavior using a time series model.

13. The method of claim 11 wherein the remedial measures further comprise sending a notification message.

14. The method of claim 11 wherein the recording user behavior comprises:

determining a value representing the user behavior.

15. The method of claim 14 wherein the plurality of items comprises search results, and

wherein the value comprises at least one of a percentage of times that a high ranking search result is selected, an average first click position, a percentage of searches that had long clicks, a percentage of searches that did not have another search within a first period of time, or a percentage of searches that did not have a reformulated search with a second period of time.