System for expanding image search using attributes and associations

• US 10,191,921 B1
• Filed: 04/03/2018
• Issued: 01/29/2019
• Est. Priority Date: 04/03/2018
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Can, Ethem F., Crowell, Richard Welland, Leeman-Munk, Samuel Paul, Peterson, Jared, Sethi, Saratendu
• Primary Examiner(s): Lee, Wilson
• Application Number:
  US15/944,163
• Time in Patent Office: 301 Days
• Field of Search: None
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

A system provides image search results based on a query that includes an attribute or an association and a concept identifier. The query is input into a trained query model to define a search syntax for the query. The search syntax is submitted to an expanded annotated image database that includes a concept image of a concept identified by the concept identifier with a plurality of attributes associated with the concept and a plurality of associations associated with the concept. A query result is received based on matching the defined search syntax to one or more of the attributes or one or more of the associations. The query result includes the concept image of the concept associated with the matched one or more of the attributes or one or more of the associations. The concept image included in the received query result is presented in a display.

Claims

1. A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device cause the computing device to:

receive a query, wherein the query is a request to identify an image relevant to the query, wherein the query includes a concept identifier and an attribute or an association;

input the received query into a trained query model to define a search syntax for the received query;

submit the defined search syntax to an expanded annotated image database, wherein the expanded annotated image database is an annotated image database combined with an attribute and association database, wherein the expanded annotated image database includes a concept image of a concept identified by the concept identifier with a plurality of attributes associated with the concept and a plurality of associations associated with the concept, wherein the attribute and association database includes a plurality of data records, wherein each data record of the plurality of data records includes a concept name, the plurality of attributes associated with the concept name, and the plurality of associations associated with the concept name, wherein an attribute describes a characteristic of the concept, wherein an association describes a relationship between two concepts;

receive a query result of the received query based on matching the defined search syntax to one or more of the plurality of attributes or one or more of the plurality of associations, wherein the query result includes the concept image of the concept associated with the matched one or more of the plurality of attributes or one or more of the plurality of associations; and

present the concept image included in the received query result in a display.

2. The non-transitory computer-readable medium of claim 1, wherein the trained query model is a trained recurrent neural network model.

3. The non-transitory computer-readable medium of claim 2, wherein the recurrent neural network model is trained with a plurality of queries and a plurality of search syntax examples.

4. The non-transitory computer-readable medium of claim 3, wherein the plurality of search syntax examples provide a vocabulary for the concept.

5. The non-transitory computer-readable medium of claim 1, further comprising computer-readable instructions that when executed by the computing device cause the computing device to, before receiving the query:

(a) select a first image from a plurality of images;

(b) split the selected first image into a plurality of images;

(c) input the selected first image and the plurality of images into an image annotation model to define a result set for each of the selected first image and the plurality of images, where each result set includes a plurality of result pairs, wherein each result pair of the plurality of result pairs includes a probability value and an image label;

(d) apply maximum pooling to the defined result set for each of the selected first image and the plurality of images to define a single result set;

(e) sort the plurality of result pairs of the defined single result set based on the probability value;

(f) filter the sorted plurality of result pairs of the defined single result set based on the probability value;

(g) output the filtered result pairs of the defined single result set to the annotated image database with an indicator of the selected first image; and

repeat (a) to (g) for each image of the plurality of images.

6. The non-transitory computer-readable medium of claim 5, wherein filtering the sorted plurality of result pairs comprises:

identifying a first weak result pair when a difference between a probability value of the first weak result pair and a previous probability value of a previous result pair exceeds the probability value of the first weak result pair, wherein the previous result pair is immediately prior to the first weak result pair of the sorted plurality of result pairs; and

removing the first weak result pair and all subsequent result pairs relative to the first weak result pair from the sorted plurality of result pairs.

7. The non-transitory computer-readable medium of claim 1, wherein the annotated image database is combined with the attribute and association database using a pattern match between the concept name and the image label.

8. The non-transitory computer-readable medium of claim 5, wherein the image annotation model is a trained convolutional neural network model.

9. The non-transitory computer-readable medium of claim 1, wherein the expanded annotated image database is created by:

training a bidirectional, text recurrent neural network with a training set of word vectors to define a sentence vector;

inputting the defined sentence vector, a concept vector, and an attribute vector into a trained fully connected neural network that determines if a new attribute or a new association is valid; and

when the new attribute or the new association is valid, the new attribute or the new association is added to the attribute and association database.

10. The non-transitory computer-readable medium of claim 9, wherein the training set of word vectors are defined by a user.

11. The non-transitory computer-readable medium of claim 5, wherein the probability value indicates a likelihood that the concept identified by the image label is included in a respective image of the selected first image and the plurality of images.

12. A computing device comprising:

a processor; and

a non-transitory computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to

receive a query, wherein the query is a request to identify an image relevant to the query, wherein the query includes a concept identifier and an attribute or an association;

input the received query into a trained query model to define a search syntax for the received query;

submit the defined search syntax to an expanded annotated image database, wherein the expanded annotated image database is an annotated image database combined with an attribute and association database, wherein the expanded annotated image database includes a concept image of a concept identified by the concept identifier with a plurality of attributes associated with the concept and a plurality of associations associated with the concept, wherein the attribute and association database includes a plurality of data records, wherein each data record of the plurality of data records includes a concept name, the plurality of attributes associated with the concept name, and the plurality of associations associated with the concept name, wherein an attribute describes a characteristic of the concept, wherein an association describes a relationship between two concepts;

receive a query result of the received query based on matching the defined search syntax to one or more of the plurality of attributes or one or more of the plurality of associations, wherein the query result includes the concept image of the concept associated with the matched one or more of the plurality of attributes or one or more of the plurality of associations; and

present the concept image included in the received query result in a display.

13. The computing device of claim 12, wherein the trained query model is a trained recurrent neural network model.

14. The computing device of claim 12, further comprising computer-readable instructions that when executed by the computing device cause the computing device to, before receiving the query:

(a) select a first image from a plurality of images;

(b) split the selected first image into a plurality of images;

(c) input the selected first image and the plurality of images into an image annotation model to define a result set for each of the selected first image and the plurality of images, where each result set includes a plurality of result pairs, wherein each result pair of the plurality of result pairs includes a probability value and an image label;

(d) apply maximum pooling to the defined result set for each of the selected first image and the plurality of images to define a single result set;

(e) sort the plurality of result pairs of the defined single result set based on the probability value;

(f) filter the sorted plurality of result pairs of the defined single result set based on the probability value;

(g) output the filtered result pairs of the defined single result set to the annotated image database with an indicator of the selected first image; and

repeat (a) to (g) for each image of the plurality of images.

15. The computing device of claim 14, wherein filtering the sorted plurality of result pairs comprises:

identifying a first weak result pair when a difference between a probability value of the first weak result pair and a previous probability value of a previous result pair exceeds the probability value of the first weak result pair, wherein the previous result pair is immediately prior to the first weak result pair of the sorted plurality of result pairs; and

removing the first weak result pair and all subsequent result pairs relative to the first weak result pair from the sorted plurality of result pairs.

16. The computing device of claim 12, wherein the annotated image database is combined with the attribute and association database using a pattern match between the concept name and the image label.

17. The computing device of claim 12, wherein the expanded annotated image database is created by:

training a bidirectional, text recurrent neural network with a training set of word vectors to define a sentence vector;

inputting the defined sentence vector, a concept vector, and an attribute vector into a trained fully connected neural network that determines if a new attribute or a new association is valid; and

when the new attribute or the new association is valid, the new attribute or the new association is added to the attribute and association database.

18. The computing device of claim 13, wherein the recurrent neural network model is trained with a plurality of queries and a plurality of search syntax examples.

19. The computing device of claim 14, wherein the image annotation model is a trained convolutional neural network model.

20. A method of identifying image search results based on a query that includes an attribute or an association, the method comprising:

receiving a query by a computing device, wherein the query is a request to identify an image relevant to the query, wherein the query includes a concept identifier and an attribute or an association;

inputting, by the computing device, the received query into a trained query model to define a search syntax for the received query;

submitting, by the computing device, the defined search syntax to an expanded annotated image database, wherein the expanded annotated image database is an annotated image database combined with an attribute and association database, wherein the expanded annotated image database includes a concept image of a concept identified by the concept identifier with a plurality of attributes associated with the concept and a plurality of associations associated with the concept, wherein the attribute and association database includes a plurality of data records, wherein each data record of the plurality of data records includes a concept name, the plurality of attributes associated with the concept name, and the plurality of associations associated with the concept name, wherein an attribute describes a characteristic of the concept, wherein an association describes a relationship between two concepts;

receiving, by the computing device, a query result of the received query based on matching the defined search syntax to one or more of the plurality of attributes or one or more of the plurality of associations, wherein the query result includes the concept image of the concept associated with the matched one or more of the plurality of attributes or one or more of the plurality of associations; and

presenting, by the computing device, the concept image included in the received query result in a display.

21. The method of claim 20, wherein the trained query model is a trained recurrent neural network model.

22. The method of claim 21, wherein the recurrent neural network model is trained with a plurality of queries and a plurality of search syntax examples.

23. The method of claim 22, wherein the plurality of search syntax examples provide a vocabulary for the concept.

24. The method of claim 20, further comprising, before receiving the query:

(a) selecting, by the computing device, a first image from a plurality of images;

(b) splitting, by the computing device, the selected first image into a plurality of images;

(c) inputting, by the computing device, the selected first image and the plurality of images into an image annotation model to define a result set for each of the selected first image and the plurality of images, where each result set includes a plurality of result pairs, wherein each result pair of the plurality of result pairs includes a probability value and an image label;

(d) applying, by the computing device, maximum pooling to the defined result set for each of the selected first image and the plurality of images to define a single result set;

(e) sorting, by the computing device, the plurality of result pairs of the defined single result set based on the probability value;

(f) filtering, by the computing device, the sorted plurality of result pairs of the defined single result set based on the probability value;

(g) outputting, by the computing device, the filtered result pairs of the defined single result set to the annotated image database with an indicator of the selected first image; and

repeating, by the computing device, (a) to (g) for each image of the plurality of images.

25. The method of claim 24, wherein filtering the sorted plurality of result pairs comprises:

identifying a first weak result pair when a difference between a probability value of the first weak result pair and a previous probability value of a previous result pair exceeds the probability value of the first weak result pair, wherein the previous result pair is immediately prior to the first weak result pair of the sorted plurality of result pairs; and

removing the first weak result pair and all subsequent result pairs relative to the first weak result pair from the sorted plurality of result pairs.

26. The method of claim 20, wherein the annotated image database is combined with the attribute and association database using a pattern match between the concept name and the image label.

27. The method of claim 24, wherein the image annotation model is a trained convolutional neural network model.

28. The method of claim 20, wherein the expanded annotated image database is created by:

training a bidirectional, text recurrent neural network with a training set of word vectors to define a sentence vector;

inputting the defined sentence vector, a concept vector, and an attribute vector into a trained fully connected neural network that determines if a new attribute or a new association is valid; and

when the new attribute or the new association is valid, the new attribute or the new association is added to the attribute and association database.

29. The method of claim 28, wherein the training set of word vectors are defined by a user.

30. The method of claim 24, wherein the probability value indicates a likelihood that the concept identified by the image label is included in a respective image of the selected first image and the plurality of images.