Moderation of user-generated content

• US 10,095,980 B1
• Filed: 02/25/2014
• Issued: 10/09/2018
• Est. Priority Date: 04/29/2011
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kulshreshtha, Ashutosh, de Alfaro, Luca, Slep, Mitchell, Cornea, Nicu D., Subramanian, Sowmya, Russell, Ethan G.
• Primary Examiner(s): Yen, Syling
• Assistant Examiner(s): TRAN, BAO G
• Application Number:
  US14/189,937
• Time in Patent Office: 1,687 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06N 5/048 Fuzzy inferencing
• CPC Class Codes: G09B 29/106 using electronic means
• CPC Class Codes: H04W 12/40 Security arrangements using identity modules

Abstract

A system and method for updating and correcting facts that receives proposed values for facts from users and determines a correctness score which is used to automatically accept or reject the proposed values.

Claims

1. A method executed by a computer system for determining a value for an attribute of a map feature, the method comprising:

receiving, from a source, a proposed value for the attribute of the map feature;

determining, by a reliability engine, for the source, a first score indicative of a probability that the source provides intentionally incorrect values, the first score based on a plurality of stored interactions of the source with a map editing service;

determining, by an accuracy engine, for the source, a second score indicative of a probability that the source provides accurate values;

determining, by the reliability engine, for the proposed value, a third score indicative of a probability that the proposed value is an intentionally incorrect value;

determining, by a consensus engine, for the proposed value, a fourth score indicative of the probability that the proposed value is correct based on the first, second and third scores;

determining, by an automoderation engine, a popularity of the map feature;

responsive to the fourth score exceeding a threshold determined responsive to the popularity of the map feature, accepting, by the automoderation engine, the proposed value for the attribute of the map feature, wherein the threshold is higher if the popularity of the map feature is higher;

identifying, by an overclustering engine, one or more characteristics indicative of overclustering with respect to the map feature, the one or more characteristics including a plurality of accepted proposed values for the attribute of the map feature; and

determining by the overclustering engine, a priority associated with the map feature for correcting the overclustering, based at least in part on the one or more characteristics.

2. The method of claim 1 wherein determining the first score comprises:

receiving the plurality of interactions of the source with a map editing service;

determining characteristics of the plurality of interactions of the source; and

determining the first score based at least in part on the determined characteristics of the plurality of interactions.

3. The method of claim 2 further comprising:

identifying a plurality of interaction sessions based on a timing of the plurality of interactions;

determining characteristics of the plurality of interaction sessions; and

determining the first score further based at least in part on the characteristics of the plurality of interaction sessions.

4. The method of claim 2 further comprising:

receiving a second plurality of interactions of the source with an online map hosting service; and

identifying types of interactions in the second plurality of interactions of the source;

wherein the first score is further based on the identified types of interactions in the second plurality of interactions.

5. The method of claim 1 wherein the plurality of stored interactions of the source with a map editing service comprise a plurality of proposed edits to map features and determining the first score comprises:

receiving the plurality of proposed edits to map features;

determining characteristics of the plurality of proposed edits; and

determining the first score based at least in part on the determined characteristics of the plurality of proposed edits.

6. The method of claim 5 further comprising:

determining from the plurality of proposed edits a plurality of edit sessions; and

wherein determining characteristics of the plurality of proposed edits comprises determining an elapsed time between proposed edits within one of the plurality of edit sessions.

7. A system for determining a value for an attribute of a map feature, the system implemented on one or more processors and comprising:

a network interface configured to receive from a source a proposed value for the attribute of the map feature;

a reliability engine configured to determine for the source a first score indicative of a probability that the source provides intentionally incorrect values, the first score based on a plurality of stored interactions of the source with a map editing service;

an accuracy engine configured to determine for the source a second score indicative of a probability that the source provides accurate values;

the reliability engine further configured to determine for the proposed value a third score indicative of a probability that the proposed value is an intentionally incorrect value;

a consensus engine configured to determine for the proposed value a fourth score indicative of the probability that the proposed value is correct based on the first, second and third scores;

an automoderation engine configured to determine a popularity of the map feature;

the automoderation engine further configured to, responsive to the fourth score exceeding a threshold determined responsive to the popularity of the map feature, accept the proposed value for the attribute of the map feature;

an overclustering engine configured to identify one or more characteristics indicative of overclustering with respect to the map feature, the one or more characteristics including a plurality of accepted proposed values for the attribute of the map feature; and

the overclustering engine further configured to determine a priority associated with the map feature for correcting the overclustering, based at least in part on the one or more characteristics.

8. The system of claim 7 wherein determining the first score comprises:

the network interface configured to receive the plurality of interactions of the source with a map editing service; and

the reliability engine further configured to;

determine characteristics of the plurality of interactions of the source; and

determine the first score based at least in part on the determined characteristics of the plurality of interactions.

12. The system of claim 7 wherein the plurality of stored interactions of the source with a map editing service comprise a plurality of proposed edits to map features and determining the first score comprises:

the network interface configured to receive the plurality of proposed edits to map features; and

the reliability engine further configured to;

determine characteristics of the plurality of proposed edits; and

determine the first score based at least in part on the determined characteristics of the plurality of proposed edits.

13. The system of claim 12 wherein the network interface configured to receive the plurality of proposed edits to a map feature comprises the network interface configured to retrieve the plurality of proposed edits to the map feature.

14. The system of claim 12 wherein:

the reliability engine is further configured to determine from the plurality of proposed edits a plurality of edit sessions; and

determining characteristics of the plurality of proposed edits comprises determining an elapsed time between proposed edits within one of the plurality of edit sessions.

9. The system of 8 wherein the network interface configured to receive the plurality of interactions comprises the network interface configured to retrieve the plurality of interactions.

10. The system of 8, wherein the reliability engine is further configured to:

identify a plurality of interaction sessions based on a timing of the plurality of interactions;

determine characteristics of the plurality of interaction sessions; and

determine the first score further based at least in part on the characteristics of the plurality of interaction sessions.

11. The system of 8 wherein:

the network interface is further configured to receive a second plurality of interactions of the source with an online map hosting service; and

the accuracy engine is further configured to;

identify types of interactions in the second plurality of interactions of the source;

wherein the second score is further based on the identified types of interactions in the second plurality of interactions.

15. A computer program product, comprising:

a non-transitory machine-readable storage medium having computer readable program code embodied therein for determining a value for an attribute of a map feature, the medium comprising computer readable program code executable to;

receive from a source a proposed value for the attribute of the map feature;

determine, by a reliability engine, for the source, a first score indicative of a probability that the source provides intentionally incorrect values, the first score based on a plurality of stored interactions of the source with a map editing service;

determine, by an accuracy engine, for the source, a second score indicative of a probability that the source provides accurate values;

determine, by the reliability engine, for the proposed value, a third score indicative of a probability that the proposed value is an intentionally incorrect value;

determine, by a consensus engine, for the proposed value, a fourth score indicative of the probability that the proposed value is correct based on the first, second and third scores;

determine, by an automoderation engine, a popularity of the map feature;

responsive to the fourth score exceeding a threshold determined responsive to the popularity of the map feature, accept, by the automoderation engine, the proposed value for the attribute of the map feature, wherein the threshold is higher if the popularity of the map feature is higher;

identify, by an overclustering engine, one or more characteristics indicative of overclustering with respect to the map feature, the one or more characteristics including a plurality of accepted proposed values for the attribute of the map feature; and

determine, by the overclustering engine, a priority associated with the map feature for correcting the overclustering, based at least in part on the one or more characteristics.

16. The computer program product of claim 15 wherein program code to determine a first score comprises program code to:

receive the plurality of interactions of the source with a map editing service;

determine characteristics of the plurality of interactions of the source; and

determine the first score based at least in part on the determined characteristics of the plurality of interactions.

17. The computer program product of claim 16 further comprising program code to:

identify a plurality of interaction sessions based on a timing of the plurality of interactions;

determine characteristics of the plurality of interaction sessions; and

determine the first score further based at least in part on the characteristics of the plurality of interaction sessions.

18. The computer program product of claim 16 further comprising program code to:

receive a second plurality of interactions of the source with an online map hosting service;

identify types of interactions in the second plurality of interactions of the source; and

wherein the second score is further based on the identified types of interactions in the second plurality of interactions.

19. The computer program product of claim 15 wherein the plurality of stored interactions of the source with a map editing service comprise a plurality of proposed edits to map features and program code to determine the first score comprises program code to:

receive the plurality of proposed edits to map features;

determine characteristics of the plurality of proposed edits; and

determine the first score based at least in part on the determined characteristics of the plurality of proposed edits.

20. The computer program product of claim 19 further comprising program code to determine from the plurality of proposed edits a plurality of edit sessions and wherein the program code to determine characteristics of the plurality of proposed edits comprises program code to determine an elapsed time between proposed edits within one of the plurality of edit sessions.