Predicting accuracy of submitted data

• US 10,223,637 B1
• Filed: 05/30/2013
• Issued: 03/05/2019
• Est. Priority Date: 05/30/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Czuba, Krzysztof, Gabrilovich, Evgeniy
• Primary Examiner(s): Rifkin, Ben M
• Application Number:
  US13/906,220
• Time in Patent Office: 2,105 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting the accuracy of user submissions. One of the methods includes receiving, from a user, an update to an attribute of an entity related to a topic. If the user is determined to be reliable relative to the topic based on user profile data of the user, the knowledge base is updated with the update to the attribute of the entity.

Claims

1. A computer-implemented method comprising:

maintaining a knowledge base accessible by multiple users, wherein the knowledge base comprises information about entities, the information about each entity being represented as one or more attribute-value pairs, each entity being assigned a unique identifier, the entities including entities having text string aliases, wherein the text string aliases include text string aliases that are shared by multiple entities;

receiving, by a search system from a user having user profile data relating specifically to the user, a search request related to a topic, wherein the user profile data is not stored in the knowledge base;

determining, from the user profile data not stored in the knowledge base, that updates to the knowledge base provided by the user for the topic are likely to be accurate;

in response, selecting, by the search system, a knowledge base attribute of an entity maintained in the knowledge base and related to the topic of the search request;

providing to the user a response to the search request received from the user, wherein the response to the search request includes a knowledge panel that presents one or more items of information about the entity and requests that the user submit, through a user interface control of the knowledge panel, an updated value for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user, wherein providing to the user the response to the search request includes providing one or more search results satisfying the search request received from the user;

receiving, from the user, an updated value for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user; and

updating the knowledge base with the updated value received from the user for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user.

2. The method of claim 1, wherein determining, from the user profile data, that updates to the knowledge base provided by the user for the topic are likely to be accurate comprises:

computing, using the user profile data as input to a user model, a likelihood that an update from the user to an entity related to the topic is accurate; and

determining that the computed likelihood satisfies a threshold.

3. The method of claim 2, wherein the user model is trained using training examples that represent previously submitted updates to the knowledge base by users, the training examples including data representing whether the previously submitted updates were accurate.

4. The method of claim 3, wherein each training example includes information from a user profile of a user that submitted an update represented by the training example.

5. The method of claim 4, wherein the information from the user profile includes one or more statistics describing a measure of accuracy of knowledge base submissions by the user.

6. The method of claim 5, wherein the information from the user profile includes information about subsystems of the search system accessed by the user, wherein the search system considers users who access more subsystems of the search system to be more reliable than users who access fewer subsystems of the search system.

7. The method of claim 2, further comprising whenever the likelihood satisfies a threshold for a particular user, updating the knowledge base with a value of the attribute provided by the particular user without further inspection by knowledge base administrators.

8. The method of claim 1, wherein updating the knowledge base with the updated value for the selected attribute of the entity comprises updating an attribute-value pair in the knowledge base without receiving confirmation by knowledge base administrators for the update.

9. The method of claim 8, further comprising:

receiving, from a different second user, a subsequent search request related to the topic; and

providing, to the second user, information about the entity including the updated value for the attribute of the entity.

10. The method of claim 1, wherein the entity is a person, a place, an animal, an event, an organization, a sports team, a movie, a song, or a work of art.

11. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

maintaining a knowledge base accessible by multiple users, wherein the knowledge base comprises information about entities, the information about each entity being represented as one or more attribute-value pairs, each entity being assigned a unique identifier, the entities including entities having text string aliases, wherein the text string aliases include text string aliases that are shared by multiple entities;

receiving, by a search system from a user having user profile data relating specifically to the user, a search request related to a topic, wherein the user profile data is not stored in the knowledge base;

determining, from the user profile data not stored in the knowledge base, that updates to the knowledge base provided by the user for the topic are likely to be accurate;

in response, selecting, by the search system, a knowledge base attribute of an entity maintained in the knowledge base and related to the topic of the search request;

providing to the user a response to the search request received from the user, wherein the response to the search request includes a knowledge panel that presents one or more items of information about the entity and requests that the user submit, through a user interface control of the knowledge panel, an updated value for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user, wherein providing to the user the response to the search request includes providing one or more search results satisfying the search request received from the user;

receiving, from the user, an updated value for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user; and

updating the knowledge base with the updated value received from the user for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user.

12. The system of claim 11, wherein determining, from the user profile data, that updates to the knowledge base provided by the user for the topic are likely to be accurate comprises:

computing, using the user profile data as input to a user model, a likelihood that an update from the user to an entity related to the topic is accurate; and

determining that the computed likelihood satisfies a threshold.

13. The system of claim 12, wherein the user model is trained using training examples that represent previously submitted updates to the knowledge base by users, the training examples including data representing whether the previously submitted updates were accurate.

14. The system of claim 13, wherein each training example includes information from a user profile of a user that submitted an update represented by the training example.

15. The system of claim 14, wherein the information from the user profile includes one or more statistics describing a measure of accuracy of knowledge base submissions by the user.

16. The system of claim 15, wherein the information from the user profile includes information about subsystems of the search system accessed by the user, wherein the search system considers users who access more subsystems of the search system to be more reliable than users who access fewer subsystems of the search system.

17. The system of claim 11, wherein updating the knowledge base with the updated value for the selected attribute of the entity comprises updating an attribute-value pair in the knowledge base without receiving confirmation by knowledge base administrators for the update.

18. The system of claim 17, wherein the operations further comprise:

receiving, from a different second user, a subsequent search request related to the topic; and

providing, to the second user, information about the entity including the updated value for the attribute of the entity.

19. The system of claim 11, wherein the entity is a person, a place, an animal, an event, an organization, a sports team, a movie, a song, or a work of art.

20. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

maintaining a knowledge base accessible by multiple users, wherein the knowledge base comprises information about entities, the information about each entity being represented as one or more attribute-value pairs, each entity being assigned a unique identifier, the entities including entities having text string aliases, wherein the text string aliases include text string aliases that are shared by multiple entities;

receiving, by a search system from a user having user profile data relating specifically to the user, a search request related to a topic, wherein the user profile data is not stored in the knowledge base;

determining, from the user profile data not stored in the knowledge base, that updates to the knowledge base provided by the user for the topic are likely to be accurate;

in response, selecting, by the search system, a knowledge base attribute of an entity maintained in the knowledge base and related to the topic of the search request;

providing to the user a response to the search request received from the user, wherein the response to the search request includes a knowledge panel that presents one or more items of information about the entity and requests that the user submit, through a user interface control of the knowledge panel, an updated value for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user, wherein providing to the user the response to the search request includes providing one or more search results satisfying the search request received from the user;

receiving, from the user, an updated value for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user; and

updating the knowledge base with the updated value received from the user for the knowledge base attribute selected by the search system for the entity maintained in the knowledge base and related to the topic of the search request received from the user.

21. The computer program product of claim 20, wherein determining, from the user profile data, that updates to the knowledge base provided by the user for the topic are likely to be accurate comprises:

computing, using the user profile data as input to a user model, a likelihood that an update from the user to an entity related to the topic is accurate; and

determining that the computed likelihood satisfies a threshold.

22. The computer program product of claim 21, wherein the operations further comprise:

whenever the likelihood satisfies a threshold for a particular user, updating the knowledge base with a value of the attribute provided by the particular user without further inspection by knowledge base administrators.

23. The computer program product of claim 20, wherein the entity is a person, a place, an animal, an event, an organization, a sports team, a movie, a song, or a work of art.