Systems and Methods for Automatically Classifying Businesses from Images

• US 20170109615A1
• Filed: 10/16/2015
• Published: 04/20/2017
• Est. Priority Date: 10/16/2015
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Movshovitz-Attias, Yair, Yatziv, Liron, Yu, Qian, Stumpe, Martin Christian, Shet, Vinay Damodar, Arnoud, Sacha Christophe
• Application Number:
  US14/885,452
• Publication Number:
  US 20170109615A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Computer-implemented methods and systems for automatically classifying businesses from imagery can include providing one or more images of a location entity as input to a statistical model that can be applied to each image. A plurality of classification labels for the location entity in the one or more images can be generated and provided as an output of the statistical model. The plurality of classification labels can be generated by selecting from an ontology that identifies predetermined relationships between location entities and categories associated with corresponding classification labels at multiple levels of granularity. Confidence scores for the plurality of classification labels can be generated to indicate a likelihood level that each generated classification label is accurate for its corresponding location entity. Associations based on the classification labels generated for each image can be stored in a database and used to help retrieve relevant business information requested by a user.

Claims

1. A computer-implemented method of providing classification labels for location entities from imagery, comprising:

providing, using one or more computing devices, one or more images of a location entity as input to a statistical model;

applying, using the one or more computing devices, the statistical model to the one or more images;

generating, using the one or more computing devices, a plurality of classification labels for the location entity in the one or more images, wherein the plurality of classification labels are generated by selecting from an ontology that identifies predetermined relationships between location entities and categories associated with corresponding classification labels at multiple levels of granularity; and

providing, using the one or more computing devices, the plurality of classification labels as an output of the statistical model.

2. The computer-implemented method of claim 1, further comprising storing in a database, using the one or more computing devices, an association between the location entity associated with the one or more images and the plurality of generated classification labels.

3. The computer-implemented method of claim 2, wherein the location entity comprises a business and wherein the database comprises business information for the location entity as well as the association between the business associated with the one or more images and the plurality of generated classification labels.

4. The computer-implemented method of claim 3, further comprising:

receiving, using the one or more computing devices, a request from a user for business information; and

retrieving, using the one or more computing devices, the requested business information from the database including the stored associations between the business associated with the one or more images and the plurality of generated classification labels.

5. The computer-implemented method of claim 3, further comprising matching, using the one or more computing devices, the one or more images to an existing business in the database using the plurality of classification labels generated for the one or more images at least in part to perform the matching.

6. The computer-implemented method of claim 1, further comprising applying, using the one or more computing devices, a bounding box to the one or more images, wherein the bounding box identifies at least one portion of the one or more images containing entity information related to the location entity, and wherein the identified at least one portion of the one or more images is provided as the input to the statistical model.

7. The computer-implemented method of claim 1, further comprising training, using the one or more computing devices, the statistical model using a set of training images of different location entities and data identifying the geographic location of the location entities within the training images, the statistical model outputting a plurality of classification labels for each training image.

8. The computer-implemented method of claim 1, further comprising generating, using the one or more computing devices, a confidence score for each of the plurality of classification labels for the location entity identified in the one or more images, wherein each confidence score indicates a likelihood level that each generated classification label is accurate for its corresponding location entity.

9. The computer-implemented method of claim 1, wherein the plurality of classification labels include at least one classification label from a first hierarchical level of categorization and at least one classification label from a second hierarchical level of categorization.

10. The computer-implemented method of claim 1, wherein the plurality of classification labels for the location entity comprises at least one classification label from a general level of categorization, the general level of categorization including one or more of an entertainment and recreation label, a health and beauty label, a lodging label, a nightlife label, a professional services label, a food and drink label and a shopping label.

11. The computer-implemented method of claim 1, further comprising tagging, using the one or more computing devices, the one or more images with the plurality of classification labels identified for the location entity in the one or more images.

12. The computer-implemented method of claim 1, wherein the location entity comprises a business.

13. The computer-implemented method of claim 1, wherein the one or more images comprise panoramic street-level images of the location entity.

14. The computer-implemented method of claim 1, wherein the statistical model is a neural network.

15. The computer-implemented method of claim 1, wherein the statistical model is a deep convolutional neural network with a logistic regression top layer.

16. A computer-implemented method of processing a business-related search query, comprising:

receiving, using one or more computing devices, a request for listing information for a particular type of business;

accessing, using the one or more computing devices, a database of business listings that comprises businesses, images of the businesses, and associations between the businesses and multiple classification labels;

wherein the associations between the businesses and multiple classification labels are identified by providing each image of a business as input to a statistical model, applying the statistical model to each image of the business, generating the multiple classification labels for the business, and providing the multiple classification labels for the business as output of the statistical model; and

providing, using the one or more computing devices, listing information including one or more business listings identified from the database of business listings at least in part by consulting the associations between the businesses and multiple classification labels.

17. The computer-implemented method of claim 16, wherein the multiple classification labels include at least one classification label from a first hierarchical level of categorization and at least one classification label from a second hierarchical level of categorization.

18. A computing device, comprising:

one or more processors; and

one or more memory devices, the one or more memory devices storing computer-readable instructions that when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

providing one or more images of a location entity an input to a statistical model;

applying the statistical model to the one or more images;

generating a plurality of classification labels for the location entity in the one or more images, wherein the plurality of classification labels are generated by selecting from an ontology that identifies predetermined relationships between location entities and categories associated with corresponding classification labels at multiple levels of granularity; and

providing the plurality of classification labels as an output of the statistical model.

19. The computing device of claim 18, wherein the operations further comprise generating a confidence score for each of the plurality of classification labels for the location entity identified in the one or more images, wherein each confidence score indicates a likelihood level that each generated classification label is accurate for its corresponding location entity.

20. The computing device of claim 18, wherein the location entity comprises a business and wherein the operations further comprise:

storing in a database an association between the business associated with the one or more images and the plurality of generated classification labels;

receiving a request from a user for business information; and

retrieving the requested business information from the database including the stored associations between the business associated with the one or more images and the plurality of generated classification labels.