Image entity recognition and response

• US 10,402,446 B2
• Filed: 04/29/2015
• Issued: 09/03/2019
• Est. Priority Date: 04/29/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kamath, Udyavara Srikanth, Zhang, Minghua, Wu, Ming, Bindlish, Ankush
• Primary Examiner(s): Stevens, Robert
• Application Number:
  US14/700,027
• Publication Number:
  US 20160321300A1
• Time in Patent Office: 1,588 Days
• Field of Search: 707769
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 Clustering techniques

Abstract

Systems and methods for responding to an image query from a computer user are provided. According to the disclosed subject matter, in response to receiving an image query, a search engine identifies the subject matter of the query image according to similar images. An entity name is determined from the similar images and is mapped to a known entity of the search engine. Based on the known entity, related information regarding the known entity is obtained and one or more search results pages directed to the known entity are generated. At least one of the generated search results pages is returned to the computer user as a response to the image query.

Claims

1. A computer implemented method operating on a computer and responding to an image query from a computer user, the method comprising:

receiving an image query, the image query including a query image of subject matter for which additional information is sought;

generating a feature vector from the query image;

identifying a set of images based on the feature vector from the query image;

obtaining image metadata from at least some of the images of the set of images;

performing clustering techniques by the computer on the image metadata from at least some of the images of the set of images to identify relevant entities and determine an entity name for the subject matter of the query image, wherein an assigned likelihood value that the entity name is correct is above a predetermined threshold;

using the entity name to select a known entity from a corpus of known entities indexed by the search engine;

identifying various sources of information regarding the known entity and obtaining additional information regarding the known entity from those various sources;

generating one or more search results pages according to the obtained additional information regarding the known entity; and

returning at least one search results page to the computer user.

2. The method of claim 1, wherein the image metadata comprises captions associated with the at least some of the images of the set of images.

3. The method of claim 1, wherein the image metadata comprises titles of associated with the at least some of the images of the set of images.

4. The method of claim 1, wherein the image metadata comprises descriptive content of the at least some of the images of the set of images.

5. The method of claim 1, wherein the image metadata comprises at least two captions associated with the at least some of the images of the set of images, titles of associated with the at least some of the images of the set of images, and descriptive content of the at least some of the images of the set of images.

6. The method of claim 1, further comprising generating one or more search results to include user actionable controls for providing feedback of a computer user regarding whether or not the known entity is the correct known entity for the image query.

7. A computer readable medium bearing computer executable instructions which, in execution on a computing system by a processor, carry out a method responding to an image query from a computer user, the method comprising:

receiving an image query, the image query including a query image of subject matter for which additional information is sought;

generating a feature vector from the query image;

identifying a set of images based on the feature vector from the query image;

performing clustering techniques by the computer system on the images of the set of images to identify relevant entities and determine an entity name for the subject matter of the query image, wherein an assigned likelihood value that the entity name is correct is above a predetermined threshold;

using the entity name to select a known entity from a corpus of known entities indexed by the search engine;

identifying various sources of information regarding the known entity and obtaining additional information regarding the known entity from those various sources;

generating one or more search results pages according to the obtained additional information regarding the known entity; and

returning at least one search results page to the computer user.

8. The computer readable media of claim 7, wherein the method further comprises generating one or more search results to include user actionable controls for providing feedback of a computer user indication regarding whether or not the known entity is the correct known entity for the image query.

9. The computer readable media of claim 8, the method further comprising:

obtaining image metadata from at least some of the images of the set of images; and

wherein performing clustering techniques to determine the entity name for the subject matter of the query image comprises conducting an analysis of the image metadata to determine a the entity name for the subject matter of the query image.

10. The computer readable media of claim 9, wherein the image metadata comprises captions associated with the at least some of the images of the set of images.

11. The computer readable media of claim 10, wherein the image metadata further comprises titles associated with the at least some of the images of the set of images.

12. The computer readable media of claim 11, wherein the image metadata further comprises descriptive content of the at least some of the images of the set of images.

13. The computer readable media of claim 9, wherein the image metadata comprises at least two captions associated with the at least some of the images of the set of images, titles of associated with the at least some of the images of the set of images, and descriptive content of the at least some of the images of the set of images.

14. A computer system providing the services of a search engine responding to an image query from a computer user, the computer system comprising a processor and a memory, wherein the processor executes instructions stored in the memory as part of or in conjunction with additional components to respond to an image query from the computer user, the image query comprising a query image of subject matter for which information is sought, the additional components comprising:

a similar images identification component configured to generate a feature vector from a query image and identify a set of images based on the feature vector from the query image;

an entity name identifier component configured to identify relevant entities and determine an entity name for the subject matter of the query image according to the set of images identified by the similar images identification component, wherein an assigned likelihood value that the entity name is correct is above a predetermined threshold;

an entity match component configured to match the entity name for the subject matter of the image query to a known entity from a corpus of known entities indexed by the search engine;

a search results retrieval component configured to identify various sources of information regarding the known entity and obtain information regarding the known entity from those various sources; and

a search results page generator configured to generate one or more search results pages according to the obtained information regarding the known entity and at least some images of the set of images, and provide at least one generated search results page to the computer user.

15. The computing system of claim 14, wherein:

the entity name identifier component obtains image metadata corresponding to the set of images identified by the images identification component; and

wherein the entity name identifier component determines an entity name for the subject matter of the query image according to the image metadata corresponding to the set of images identified by the similar images identification component.

16. The computing system of claim 15, wherein the image metadata comprises captions associated with the at least some of the images of the set of images.

17. The computing system of claim 15, wherein the image metadata further comprises titles associated with the at least some of the images of the set of images.

18. The computing system of claim 15, wherein the image metadata comprises at least two captions associated with the at least some of the images of the set of images, titles of associated with the at least some of the images of the set of images, and descriptive content of the at least some of the images of the set of images.

19. The computing system of claim 15, wherein the search results page generator is configured to generate one or more search results to include user actionable controls for providing feedback of a computer user indication regarding whether or not the known entity is the correct known entity for the image query.