Systems and Methods for Identifying Entities Directly from Imagery

• US 20170286805A1
• Filed: 04/04/2016
• Published: 10/05/2017
• Est. Priority Date: 04/04/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Yu, Qian, Yatziv, Liron, Li, Yeqing, Szegedy, Christian, Arnoud, Sacha Christopher, Stumpe, Martin C.
• Granted Patent: US 10,007,867 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Systems and methods of identifying entities are disclosed. In particular, one or more images that depict an entity can be identified from a plurality of images. One or more candidate entity profiles can be determined from an entity directory based at least in part on the one or more images that depict the entity. The one or more images that depict the entity and the one or more candidate entity profiles can be provided as input to a machine learning model. One or more outputs of the machine learning model can be generated. Each output can include a match score associated with an image that depicts the entity and at least one candidate entity profile. The entity directory can be updated based at least in part on the one or more generated outputs of the machine learning model.

Claims

1. A computer-implemented method of identifying an entity, the method comprising:

identifying, by one or more computing devices, from a plurality of images, one or more images that depict an entity;

determining, by the one or more computing devices, one or more candidate entity profiles from an entity directory based at least in part on the one or more images that depict the entity;

providing, by the one or more computing devices, the one or more images that depict the entity and the one or more candidate entity profiles as input to a machine learning model;

generating, by the one or more computing devices, one or more outputs of the machine learning model, each output comprising a match score associated with an image that depicts the entity and at least one candidate entity profile; and

updating, by the one or more computing devices, the entity directory based at least in part on the one or more generated outputs of the machine learning model.

2. The computer-implemented method of claim 1, further comprising:

receiving, by the one or more computing devices, from a user device, data indicative of a user request to access information associated with the entity; and

providing, by the one or more computing devices, the requested information to the user device based at least in part on the updated entity directory.

3. The computer-implemented method of claim 1, wherein at least one match score is generated without explicitly transcribing text depicted in the corresponding image.

4. The computer-implemented method of claim 1, wherein each match score provides a degree of confidence that an entity depicted in an image corresponds to a candidate entity profile.

5. The computer-implemented method of claim 1, further comprising:

comparing, by the one or more computing devices, each match score to a match threshold; and

when the match score associated with an image that depicts the entity and a candidate entity profile is greater than the match threshold, associating, by the one or more computing devices, the entity with the candidate entity profile.

6. The computer-implemented method of claim 5, further comprising, when the match score associated with an image that depicts the entity and a candidate entity profile is greater than the match threshold, associating at least one of the one or more images that depict the entity with the candidate entity profile.

7. The computer-implemented method of claim 1, further comprising determining, by the one or more computing devices, location information associated with the one or more images that depict the entity.

8. The computer-implemented method of claim 7, wherein determining, by the one or more computing devices, one or more candidate entity profiles stored in an entity directory comprises:

accessing, by the one or more computing devices, the entity directory, the entity directory comprising a plurality of entity profiles; and

filtering, by the one or more computing devices, the entity directory based at least in part on the location information associated with the one or more images that depict the entity to determine the one or more candidate entity profiles.

9. The computer-implemented method of claim 1, further comprising:

obtaining, by the one or more computing devices, a plurality of training images having verified associated entity information; and

training, by the one or more computing devices, the machine learning model based at least in part on the plurality of training images and the verified associated entity information.

10. The computer-implemented method of claim 9, wherein training, by the one or more computing devices, the machine learning model comprises adjusting the machine learning model based at least in part on the plurality of training images and the verified associated entity information.

11. The computer-implemented method of claim 1, wherein updating, by the one or more computing devices, the entity directory comprises adjusting at least one entity profile based at least in part on the one or more generated outputs to the machine learning model.

12. The computer-implemented method of claim 1, wherein updating, by the one or more computing devices, the entity directory comprises creating a new entity profile based at least in part on the one or more generated outputs to the machine learning model.

13. The computer-implemented method of claim 1, wherein the machine learning model comprises a neural network and at least one recurrent neural network.

14. The computer-implemented method of claim 13, wherein the neural network is a deep convolutional neural network, and wherein the at least one recurrent neural network is a long short-term memory network.

15. The computer-implemented method of claim 14, wherein providing, by the one or more computing devices, the one or more images that depict the entity and the one or more candidate entity profiles as input to a machine learning model comprises providing data indicative of the one or more images that depict the entity as input to the deep convolutional neural network and providing data indicative of the one or more candidate entity profiles as input to the long short-term memory network.

16. The computer-implemented method of claim 1, wherein the entity is a business, and wherein the entity profile comprises data associated with the business.

17. The computer-implemented method of claim 16, wherein the data associated with the business comprises at least one of a name associated with the business, an address associated with the business, a logo associated with the business, a phone number associated with the business, a URL associated with the business, or one or more categories associated with the business.

18. A computing system, comprising:

one or more processors; and

one or more memory devices, the one or more memory devices storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising;

identifying from a plurality of images, one or more images that depict an entity;

determining one or more candidate entity profiles from an entity directory based at least in part on the one or more images that depict the entity;

providing the one or more images that depict the entity and the one or more candidate entity profiles as input to a machine learning model;

generating one or more outputs of the machine learning model, each output comprising a match score associated with an image that depicts the entity and at least one candidate entity profile; and

updating the entity directory based at least in part on the one or more generated outputs of the machine learning model.

19. The computing system of claim 18, wherein at least one match score is generated without explicitly transcribing text depicted in the corresponding image.

20. One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:

identifying from a plurality of images, one or more images that depict an entity;

determining one or more candidate entity profiles from an entity directory based at least in part on the one or more images that depict the entity;

providing the one or more images that depict the entity and the one or more candidate entity profiles as input to a machine learning model;

generating one or more outputs of the machine learning model, each output comprising a match score associated with an image that depicts the entity and at least one candidate entity profile; and

updating the entity directory based at least in part on the one or more generated outputs of the machine learning model.