Systems and methods for identifying entities directly from imagery

• US 10,007,867 B2
• Filed: 04/04/2016
• Issued: 06/26/2018
• Est. Priority Date: 04/04/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Yu, Qian, Yatziv, Liron, Li, Yeqing, Szegedy, Christian, Arnoud, Sacha Christopher, Stumpe, Martin C.
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US15/089,677
• Publication Number:
  US 20170286805A1
• Time in Patent Office: 813 Days
• Field of Search: None
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Systems and methods of identifying entities are disclosed. In particular, one or more images that depict an entity can be identified from a plurality of images. One or more candidate entity profiles can be determined from an entity directory based at least in part on the one or more images that depict the entity. The one or more images that depict the entity and the one or more candidate entity profiles can be provided as input to a machine learning model. One or more outputs of the machine learning model can be generated. Each output can include a match score associated with an image that depicts the entity and at least one candidate entity profile. The entity directory can be updated based at least in part on the one or more generated outputs of the machine learning model.

Claims

1. A computer-implemented method comprising:

identifying, by one or more computing devices and from a plurality of images, one or more images that depict an entity;

determining, by the one or more computing devices, location information associated with the one or more images that depict the entity;

identifying, by the one or more computing devices and based at least in part on the location information associated with the one or more images that depict the entity, one or more candidate entity profiles from an entity directory;

providing, by the one or more computing devices, the one or more images that depict the entity and the one or more candidate entity profiles as input to a machine learning model comprising a neural network and at least one recurrent neural network, the neural network comprising a deep convolutional neural network (CNN), the at least one recurrent neural network comprising a long short-term memory network (LSTM), the CNN being configured to receive data indicative of the one or more images that depict the entity, extract features from the one or more images that depict the entity, and provide data indicative of the extracted features to the LSTM, the LSTM being configured to receive data indicative of the one or more candidate entity profiles, obtain text-related information from the one or more candidate entity profiles, and model at least a portion of structured information from a candidate entity profile as a sequence of characters, such that a match score between the extracted features and data from the candidate entity profile can be determined;

generating, by the one or more computing devices, one or more outputs of the machine learning model, each output comprising a match score associated with at least one candidate entity profile and an image that depicts the entity; and

updating, by the one or more computing devices, the entity directory based at least in part on the one or more generated outputs of the machine learning model.

2. The computer-implemented method of claim 1, further comprising:

receiving, by the one or more computing devices, from a user device, data indicative of a user request to access information associated with the entity; and

providing, by the one or more computing devices, the requested information to the user device based at least in part on the updated entity directory.

3. The computer-implemented method of claim 1, wherein at least one match score is generated without explicitly transcribing text depicted in a corresponding image.

4. The computer-implemented method of claim 1, wherein each match score provides a degree of confidence that an entity depicted in an image corresponds to a candidate entity profile.

5. The computer-implemented method of claim 1, further comprising;

comparing, by the one or more computing devices, each match score to a match threshold; and

when a match score associated with a particular candidate entity profile and an image that depicts the entity is greater than the match threshold, associating, by the one or more computing devices, the entity with the particular candidate entity profile.

6. The computer-implemented method of claim 5, further comprising, when the match score associated with the particular candidate entity profile and the image that depicts the entity is greater than the match threshold, associating at least one of the one or more images that depict the entity with the particular candidate entity profile.

7. The computer-implemented method of claim 1,

wherein identifying the one or more candidate entity profiles comprises:

accessing, by the one or more computing devices, the entity directory, the entity directory comprising a plurality of entity profiles; and

filtering, by the one or more computing devices, the entity directory based at least in part on the location information associated with the one or more images that depict the entity to identify the one or more candidate entity profiles.

8. The computer-implemented method of claim 1, further comprising:

obtaining, by the one or more computing devices, a plurality of training images having verified associated entity information; and

training, by the one or more computing devices, the machine learning model based at least in part on the plurality of training images and the verified associated entity information.

9. The computer-implemented method of claim 8, Wherein training, by the one or more computing devices, the machine learning model comprises adjusting the machine learning model based at least in part on the plurality of training images and the verified associated entity information.

10. The computer-implemented method of claim 1, wherein updating the entity directory comprises modifying, based at least in part on the one or more generated outputs of the machine learning model, at least one entity profile.

11. The computer-implemented method of claim 1, wherein updating the entity directory comprises creating, based at least in part on the one or more generated outputs of the machine learning model, a new entity profile.

12. The computer-implemented method of claim 1, wherein the entity is a business, and wherein the at least one candidate entity profile comprises data associated with the business.

13. The computer-implemented method of claim 12, wherein the data associated with the business comprises at least one of a name associated with the business, an address associated with the business, a logo associated with the business, a phone number associated with the business, a uniform resource locator (URL) associated with the business, or one or more categories associated with the business.

14. A computing system, comprising:

one or more processors; and

one or more memory devices storing computer-readable instructions that when executed by the one or more processors cause the computing system to perform operations comprising;

identifying, from a plurality of images, one or more images that depict an entity;

determining location information associated with the one or more images that depict the entity;

identifying, based at least in part on the location information associated with the one or more images that depict the entity, one or more candidate entity profiles from an entity directory;

providing the one or more images that depict the entity and the one or more candidate entity profiles as input to a machine learning model comprising a neural network and at least one recurrent neural network, the neural network comprising a deep convolutional neural network (CNN), the at least one recurrent neural network comprising a long short-term memory network (LSTM), the CNN being configured to receive data indicative of the one or more images that depict the entity, extract features from the one or more images that depict the entity, and provide data indicative of the extracted features to the LSTM, the LSTM being configured to receive data indicative of the one or more candidate entity profiles, obtain text-related information from the one or more candidate entity profiles, and model at least a portion of structured information from a candidate entity profile as a sequence of characters, such that a match score between the extracted features and data from the candidate entity profile can be determined;

generating one or more outputs of the machine learning model, each output comprising a match score associated with at least one candidate entity profile and an image that depicts the entity; and

updating the entity directory based at least in part on the one or more generated outputs of the machine learning model.

15. The computing system of claim 14, wherein at least one match score is generated without explicitly transcribing text depicted in a corresponding image.

17. The computing system of claim 14, wherein the entity is a business, and wherein the at least one candidate entity profile comprises data associated with the business.

16. One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations comprising:

identifying, from a plurality of images, one or more images that depict an entity;

determining location information associated with the one or more images that depict the entity;

identifying, based at least in part on the location information associated with the one or more images that depict the entity, one or more candidate entity profiles from an entity directory;

providing the one or more images that depict the entity and the one or more candidate entity profiles as input to a machine learning model comprising a neural network and at least one recurrent neural network, the neural network comprising a deep convolutional neural network (CNN), the at least one recurrent neural network comprising a long short-term memory network (LSTM), the CNN being configured to receive data indicative of the one or more images that depict the entity, extract features from the one or more images that depict the entity, and provide data indicative of the extracted features to the LSTM, the LSTM being configured to receive data indicative of the one or more candidate entity profiles, obtain text-related information from the one or more candidate entity profiles, and model at least a portion of structured information from a candidate entity profile as a sequence of characters, such that a match score between the extracted features and data from the candidate entity profile can be determined;

generating one or more outputs of the machine learning model, each output comprising a match score associated with at least one candidate entity profile and an image that depicts the entity; and

updating the entity directory based at least in part on the one or more generated outputs of the machine learning model.

18. The one or more tangible, non-transitory computer-readable media of claim 16, wherein at least one match score is generated without explicitly transcribing text depicted in a corresponding image.

19. The one or more tangible, non-transitory computer-readable media of claim 16, wherein the entity is a business, and wherein the at least one candidate entity profile comprises data associated with the business.