Clustering search results based on image composition

• US 11,042,586 B2
• Filed: 12/29/2016
• Issued: 06/22/2021
• Est. Priority Date: 12/29/2016
• Status: Active Grant

• Current Assignee: Shutterstock Incorporated
• Original Assignee: Shutterstock Incorporated
• Inventors: Hohwald, Heath, Lazare, Lawrence
• Primary Examiner(s): Syed, Farhan M
• Application Number:
  US15/394,783
• Publication Number:
  US 20180189325A1
• Time in Patent Office: 1,636 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Various aspects of the subject technology relate to systems, methods, and machine-readable media for clustering search results based on image composition. A system may, for each image in a set of images, determine a compositional vector representing one or more objects and corresponding locations within the image using a trained computer-operated convolutional neural network. The system may provide each image through a clustering algorithm to produce one or more clusters based on compositional similarity. The system may provide images from the set of images clustered by composition, in which the images include a different listing of images for each of the one or more clusters. The system may provide a prioritized listing of images responsive to a user search query, in which the prioritized listing of images includes a different listing of images for each cluster of compositional similarity based on the metadata of each image associated with the cluster.

Claims

1. A computer-implemented method, comprising:

training a computer-operated convolutional neural network to recognize an object in a region of an image as salient using feature descriptor vectors obtained from extracted features of each saliency region of a training image;

for each image in a set of images, determining a compositional vector representing one or more objects and corresponding locations within the image using the trained computer-operated convolutional neural network;

providing each image through a clustering algorithm to produce one or more clusters based on compositional similarity, wherein the clustering algorithm maps each image to a cluster representing one of a plurality of predetermined compositional classes;

providing images from the set of images clustered by composition, the images including a different listing of images for each of the one or more clusters; and

transmitting, from a server to a client device for display by the client device, a set of search results responsive to a user search query, the set of search results including a prioritized listing of the images from each cluster of compositional similarity identified for display for a respective composition.

2. The computer-implemented method of claim 1, further comprising:

receiving, from the client device, a user input identifying the user search query for content, the user input indicating one or more queries that indicate a specific composition for an image; and

determining search results that are responsive to the user search query, wherein each image processed through the trained computer-operated convolutional neural network is provided from the search results.

3. The computer-implemented method of claim 1, further comprising:

defining a distance function that determines a cosine angle difference between at

least two compositional vectors,

wherein each image is processed through the clustering algorithm using the distance function.

4. The computer-implemented method of claim 1, further comprising:

defining a distance function that determines a cosine similarity between the compositional vector representing the image and a compositional vector representing one of the plurality of predetermined compositional classes,

wherein each image is processed through the clustering algorithm using the distance function.

5. The computer-implemented method of claim 4, wherein the distance function determines a cosine angle difference between the compositional vector representing the image and the compositional vector representing the predetermined compositional class.

6. The computer-implemented method of claim 4, wherein the distance function determines a cosine distance between the compositional vector representing the image and the compositional vector representing the predetermined compositional class.

7. The computer-implemented method of claim 1, further comprising:

storing a metadata of each image in an image collection, the metadata of each image indicating a compositional class for the image,

wherein each image processed through the trained computer-operated convolutional neural network is provided from the image collection.

8. The computer-implemented method of claim 7, wherein the prioritized listing of the images includes a different listing of images for each cluster of compositional similarity based on the metadata of each image associated with the cluster.

9. The computer-implemented method of claim 1, further comprising:

determining a predetermined number of compositional classes;

for each of the predetermined compositional classes, generating an icon representation of a composition corresponding to a compositional class;

providing, for display, a user-selectable control to filter search results responsive to a user search query, the user-selectable control including one or more icon representations corresponding to respective clusters of compositional classes;

identifying a set of images that corresponds to a compositional class associated with a user-selected icon representation based on a user interaction with the user-selectable control; and

providing a prioritized listing of images from the set of images for display on the client device.

10. The computer-implemented method of claim 1, further comprising:

determining a centroid vector for each of the one or more clusters;

generating an icon representation of a composition corresponding to the centroid vector, the icon representation indicating a compositional class of the cluster;

providing, for display, a user-selectable control to filter search results responsive to a user search query, the user-selectable control including one or more icon representations corresponding to respective clusters of compositional classes;

identifying a set of images that corresponds to a compositional class associated with a user-selected icon representation based on user interaction with the user-selectable control; and

providing a prioritized listing of images from the set of images for display on the client device.

11. The computer-implemented method of claim 1, further comprising:

obtaining behavioral data from session logs associated with a user;

generating a compositional profile for the user, the compositional profile including an N-dimensional vector where each element of the N-dimensional vector indicates a probability that a next image corresponds to a compositional class associated with the element, wherein N corresponds to a number of clusters produced; and

for each image in the set of search results, applying a score based on an element from the N-dimensional vector in the compositional profile, the element corresponding to a compositional class of the image,

wherein images from the prioritized listing of the images are prioritized based on the score applied to each image, and

wherein each image processed through the trained computer-operated convolutional neural network is provided from the behavioral data.

12. The computer-implemented method of claim 1, further comprising:

providing a set of training images for each object class of a plurality of object classes to the trained computer-operated convolutional neural network;

for each object class of the plurality of object classes, training the computer-operated convolutional neural network to recognize an object in a region of an image as salient from the feature descriptor vectors; and

providing the trained computer-operated convolutional neural network to recognize salient objects with localization in images.

13. A system, comprising:

one or more processors; and

a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to;

train a computer-operated convolutional neural network to recognize an object in a region of an image as salient using feature descriptor vectors obtained from extracted features of each saliency region of a training image;

for each image in a set of images, determine a compositional vector representing one or more objects and corresponding locations within the image using the computer-operated convolutional neural network;

provide each image through a clustering algorithm to produce one or more clusters based on compositional similarity, wherein the clustering algorithm maps each image to a cluster representing one of a plurality of predetermined compositional classes;

provide images from the set of images clustered by composition, the images including a different listing of images for each of the one or more clusters;

store a metadata of each image in an image collection, the metadata of each image indicating a compositional class for the image; and

transmit, from a server to a client device for display by the client device, a prioritized listing of images responsive to a user search query, the prioritized listing of images including a different listing of images for each cluster of compositional similarity based on the metadata of each image associated with the cluster identified for display for a respective composition.

14. The system of claim 13, wherein the instructions further cause the one or more processors to:

define a distance function that determines a cosine angle difference between at least two compositional vectors,

wherein each image is processed through the clustering algorithm using the distance function.

15. The system of claim 13, wherein the instructions further cause the one or more processors to:

define a distance function that determines a cosine similarity between the compositional vector representing the image and a compositional vector representing one of the plurality of predetermined compositional classes,

wherein each image is processed through the clustering algorithm using the distance function.

16. The system of claim 15, wherein the distance function determines a cosine angle difference between the compositional vector representing the one or more objects and corresponding locations within the image and the compositional vector representing the predetermined compositional class.

17. The system of claim 13, wherein the instructions further cause the one or more processors to:

determine a predetermined number of compositional classes;

for each of the predetermined compositional classes, generate an icon representation of a composition corresponding to the compositional class;

provide, for display, a user-selectable control to filter search results responsive to a user search query, the user-selectable control including one or more icon representations corresponding to respective clusters of compositional classes;

identify a set of images that correspond to a compositional class associated with a user-selected icon representation based on user interaction with the user-selectable control; and

provide, for transmission, a prioritized listing of images from the set of images for display on the client device.

18. The system of claim 13, wherein the instructions further cause the one or more processors to:

determine a centroid vector for each of the one or more clusters;

generate an icon representation of a composition corresponding to the centroid vector, the icon representation indicating a compositional class of the cluster;

provide, for display, a user-selectable control to filter search results responsive to a user search query, the user-selectable control including one or more icon representations corresponding to respective clusters of compositional classes;

identify a set of images that corresponds to a compositional class associated with a user-selected icon representation based on user interaction with the user-selectable control; and

provide, for transmission, a prioritized listing of images from the set of images for display on a client device.

19. The system of claim 13, wherein the instructions further cause the one or more processors to:

obtain behavioral data from session logs associated with a user;

generate a compositional profile for the user, the compositional profile including an N-dimensional vector where each element of the N-dimensional vector indicates a probability that a next image corresponds to a compositional class associated with the element, wherein N corresponds to a number of clusters produced; and

for each image in the prioritized listing of images, apply a score based on an element from the N-dimensional vector in the compositional profile, the element corresponding to a compositional class of the image,

wherein images from the prioritized listing of images are prioritized based on the score applied to each image, and

wherein each image processed through the computer-operated convolutional neural network is provided from the behavioral data.

20. A computer-implemented method, comprising:

receiving, over a transmission at a server from a client device, a user input via an application on the client device to initiate an image search, the user input indicating one or more queries that define a specific composition for an image;

generating, in response to the user input, an image search query from the user input;

providing, for transmission, the image search query over a connection to the server, the server including an image search service that obtains a set of images responsive to the image search query based on a cosine similarity between a compositional vector associated with the image search query and one or more compositional vectors of corresponding images from an image collection, the image search service clustering the set of images using a trained computer-operated convolutional neural network configured to recognize an object in a region of an image as salient using feature descriptor vectors obtained from extracted features of each saliency region of a training image, and based on composition similarity using a clustering algorithm mapping each image to a cluster representing one of a plurality of predetermined compositional classes; and

receiving a set of search results responsive to the image search query from the server, the set of search results including a prioritized listing of images identified for display by the client device for a respective composition.