Image searching

• US 9,940,575 B2
• Filed: 06/04/2015
• Issued: 04/10/2018
• Est. Priority Date: 06/04/2015
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Yahoo Holdings, Inc. (Verizon Communications Inc.)
• Inventors: Hsiao, JenHao
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US14/730,476
• Publication Number:
  US 20160357748A1
• Time in Patent Office: 1,041 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: Y04S 10/50 Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications

Abstract

As provided herein, a domain model, corresponding to a domain of an image, may be merged with a pre-trained fundamental model to generate a trained fundamental model. The trained fundamental model may comprise a feature description of the image converted into a binary code. Responsive to a user submitting a search query, a coarse image search may be performed, using a search query binary code derived from the search query, to identify a candidate group, comprising one or more images, having binary codes corresponding to the search query binary code. A fine image search may be performed on the candidate group utilizing a search query feature description derived from the search query. The fine image search may be used to rank images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group.

Claims

1. A system for image searching, comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause implementation of an image searching component configured to;

output, from a fully connected layer of a pre-trained fundamental model, a feature description of an image;

identify a domain of the image;

merge a domain model, corresponding to the domain, into the pre-trained fundamental model to generate a trained fundamental model;

convert the feature description into a binary code using the trained fundamental model;

responsive to a user submitting a search query, perform a coarse image search using a search query binary code derived from the search query to identify a candidate group, comprising one or more images, having binary codes that exceed a threshold similarity to the search query binary code, the candidate group comprising the image having the binary code;

perform a fine image search on the candidate group utilizing a search query feature description derived from the search query to rank the one or more images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group, the candidate group comprising the image having the feature description; and

responsive to the image comprising a ranking above a ranking threshold, present the image to the user as a query result for the search query.

2. The system of claim 1, the trained fundamental model comprising a convolutional neural network.

3. The system of claim 1, the image searching component configured to:

create the trained fundamental model to comprise a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer.

4. The system of claim 1, the image searching component configured to:

create the trained fundamental model to comprise a first fully connected layer, a second fully connected layer, and a latent layer.

5. The system of claim 4, the image searching component configured to:

utilize the latent layer to convert a first query output, of the first fully connected layer comprising the feature description, into the binary code describing latent semantic content of the image.

6. The system of claim 5, the image searching component configured to:

utilize the second fully connected layer to encode semantic information of a second query output of the latent layer.

7. The system of claim 5, the image searching component configured to:

input the search query into the coarse image search; and

generate, using the coarse image search, the search query binary code by utilizing a second query output of the latent layer.

8. The system of claim 5, the image searching component configured to:

input the candidate group into the fine image search; and

utilize one or more first query outputs of the first fully connected layer, using the fine image search, to rank the one or more images comprised in the candidate group.

9. The system of claim 1, the image searching component configured to:

train the domain model to learn visual descriptors corresponding to the domain to generate learned descriptors; and

utilize the learned descriptors as feature descriptions for identifying query results.

10. The system of claim 9, the image searching component configured to:

transfer the learned descriptors to the trained fundamental model, from the pre-trained fundamental model, utilizing back-propagation.

11. The system of claim 1, the image searching component configured to:

train the trained fundamental model to learn binary codes from the pre-trained fundamental model and domain model.

12. A method of image searching comprising:

training a fundamental model using an image database to create a pre-trained fundamental model, the pre-trained fundamental model comprising a convolutional layer and a fully connected layer;

outputting, from the fully connected layer of the pre-trained fundamental model, a feature description of an image;

identifying a domain of the image;

merging a domain model, corresponding to the domain, with the pre-trained fundamental model to generate a trained fundamental model;

converting the feature description into a binary code using the trained fundamental model;

responsive to a user submitting a search query, performing a coarse image search using a search query binary code derived from the search query to identify a candidate group, comprising one or more images, having binary codes corresponding to the search query binary code, the candidate group comprising the image having the binary code;

performing a fine image search on the candidate group utilizing a search query feature description derived from the search query to rank the one or more images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group, the candidate group comprising the image having the feature description; and

responsive to the image comprising a rank above a ranking threshold, presenting the image to the user as a query result for the search query.

13. The method of claim 12, the training a fundamental model comprising:

training a convolutional neural network.

14. The method of claim 12, comprising:

creating the trained fundamental model to comprise a first fully connected layer, a second fully connected layer, and a latent layer.

15. The method of claim 14, comprising:

utilizing the latent layer to convert a first query output, of the first fully connected layer comprising the feature description, into the binary code describing a latent semantic content of the image; and

utilizing the second fully connected layer to encode semantic information of a second query output of the latent layer.

16. The method of claim 14, the performing a course image search comprising:

inputting the search query into the coarse image search; and

generating, using the coarse image search, the search query binary code by utilizing a second query output of the latent layer.

17. The method of claim 14, the performing a fine image search comprising:

inputting the candidate group into the fine image search; and

utilizing first query outputs of the first fully connected layer, using the fine image search, to rank the one or more images comprised in the candidate group.

18. The method of claim 12, comprising:

training the domain model to learn visual descriptors corresponding to the domain to generate learned descriptors; and

utilizing the learned descriptors as feature descriptions for identifying query results.

19. A system for image searching, comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause implementation of an image searching component configured to;

output, from a fully connected layer of a pre-trained convolutional neural network (CNN) model, a feature description of an image;

identify a domain of the image;

merge a domain model, corresponding to the domain, with the pre-trained CNN model to generate a CNN model;

convert the feature description into a binary code using the CNN model;

responsive to a user submitting a search query, perform a coarse image search using a search query binary code derived from the search query to identify a candidate group, comprising one or more images, having binary codes corresponding to the search query binary code, the candidate group comprising the image having the binary code;

perform a fine image search on the candidate group utilizing a search query feature description derived from the search query to rank the one or more images within the candidate group based upon a similarity between the search query feature description and feature descriptions of the one or more images within the candidate group, the candidate group comprising the image having the feature description; and

responsive to the image comprising a rank above a ranking threshold, present the image to the user as a query result for the search query.

20. The system of claim 19, the image searching component configured to:

create the trained CNN model to comprise a first fully connected layer, a second fully connected layer, and a latent layer;

utilize the latent layer to convert a first query output, of the first fully connected layer comprising the feature description, into the binary code describing a latent semantic content of the image; and

utilize the second fully connected layer to encode semantic information of a second query output of the latent layer.