CLUSTERING SEARCH RESULTS BASED ON IMAGE COMPOSITION

• US 20180189325A1
• Filed: 12/29/2016
• Published: 07/05/2018
• Est. Priority Date: 12/29/2016
• Status: Active Grant

• Current Assignee: Shutterstock Incorporated
• Original Assignee: Shutterstock Incorporated
• Granted Patent: US 11,042,586 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Various aspects of the subject technology relate to systems, methods, and machine-readable media for clustering search results based on image composition. A system may, for each image in a set of images, determine a compositional vector representing one or more objects and corresponding locations within the image using a trained computer-operated convolutional neural network. The system may provide each image through a clustering algorithm to produce one or more clusters based on compositional similarity. The system may provide images from the set of images clustered by composition, in which the images include a different listing of images for each of the one or more clusters. The system may provide a prioritized listing of images responsive to a user search query, in which the prioritized listing of images includes a different listing of images for each cluster of compositional similarity based on the metadata of each image associated with the cluster.

Claims

1. A computer-implemented method, comprising:

for each image in a set of images, determining a compositional vector representing one or more objects and corresponding locations within the image using a trained computer-operated convolutional neural network;

providing each image through a clustering algorithm to produce one or more clusters based on compositional similarity;

providing images from the set of images clustered by composition, the images including a different listing of images for each of the one or more clusters; and

providing a set of search results responsive to a user search query to a client device, the set of search results including a prioritized listing of the images from each cluster of compositional similarity.

2. The computer-implemented method of claim 1, further comprising:

receiving, from the client device, user input identifying the search query for content, the user input indicating one or more queries that indicate a specific composition for an image; and

determining search results that are responsive to the search query,

wherein each image processed through the trained computer-operated convolutional neural network is provided from the search results.

3. The computer-implemented method of claim 1, further comprising:

defining a distance function that determines a cosine angle difference between at least two compositional vectors,

wherein each image is processed through the clustering algorithm using the defined distance function.

4. The computer-implemented method of claim 1, further comprising:

defining a distance function that determines a cosine similarity between the compositional vector representing the image and a compositional vector representing one of a plurality of predetermined compositional classes,

wherein each image is processed through the clustering algorithm using the defined distance function, and

wherein the clustering algorithm maps each image to a cluster representing one of the plurality of predetermined compositional classes.

5. The computer-implemented method of claim 4, wherein the distance function determines a cosine angle difference between the compositional vector representing the image and the compositional vector representing the predetermined compositional class.

6. The computer-implemented method of claim 4, wherein the distance function determines a cosine distance between the compositional vector representing the image and the compositional vector representing the predetermined compositional class.

7. The computer-implemented method of claim 1, further comprising:

storing metadata with each image in an image collection, the metadata indicating a compositional class for the image,

wherein each image processed through the trained computer-operated convolutional neural network is provided from the image collection.

8. The computer-implemented method of claim 7, wherein the prioritized listing of the images includes a different listing of images for each cluster of compositional similarity based on the metadata of each image associated with the cluster.

9. The computer-implemented method of claim 1, further comprising:

determining a predetermined number of compositional classes;

for each of the predetermined compositional classes, generating an icon representation of a composition corresponding to the compositional class;

providing for display a user-selectable control to filter search results responsive to a user search query, the user-selectable control including one or more icon representations corresponding to respective clusters of compositional classes;

identifying a set of images that correspond to a compositional class associated with a user-selected icon representation based on user interaction with the user-selectable control; and

providing a prioritized listing of images from the identified set of images for display on the client device.

10. The computer-implemented method of claim 1, further comprising:

determining a centroid vector for each of the one or more clusters;

generating an icon representation of a composition corresponding to the centroid vector, the icon representation indicating a compositional class of the cluster;

providing for display a user-selectable control to filter search results responsive to a user search query, the user-selectable control including one or more icon representations corresponding to respective clusters of compositional classes;

identifying a set of images that correspond to a compositional class associated with a user-selected icon representation based on user interaction with the user-selectable control; and

providing a prioritized listing of images from the identified set of images for display on the client device.

11. The computer-implemented method of claim 1, further comprising:

obtaining behavioral data from session logs associated with a user;

generating a compositional profile for the user, the compositional profile including an N-dimensional vector where each element of the vector indicates a probability that a next image corresponds to a compositional class associated with the element, where N corresponds to a number of clusters produced; and

for each image in the set of search results, applying a score based on an element from the N-dimensional vector in the compositional profile, the element corresponding to a compositional class of the image,

wherein images from the prioritized listing of the images are prioritized based on the score applied to each image, and

wherein each image processed through the trained computer-operated convolutional neural network is provided from the obtained behavioral data.

12. The computer-implemented method of claim 1, further comprising:

providing a set of training images for each object class of a plurality of object classes to a computer-operated convolutional neural network;

for each training image, extracting features from each saliency region of the image using the computer-operated convolutional neural network;

obtaining a feature descriptor vector for the saliency region from the extracted features of the saliency region using the computer-operated convolutional neural network;

for each object class of the plurality of object classes, training the computer-operated convolutional neural network to recognize an object in a region of an image as salient from the feature descriptor vectors; and

providing the trained computer-operated convolutional neural network to recognize salient objects with localization in images.

13. A system comprising:

one or more processors;

a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to;

for each image in a set of images, determine a compositional vector representing one or more objects and corresponding locations within the image using a trained computer-operated convolutional neural network;

provide each image through a clustering algorithm to produce one or more clusters based on compositional similarity;

provide images from the set of images clustered by composition, the images including a different listing of images for each of the one or more clusters;

store metadata with each image in an image collection, the metadata indicating a compositional class for the image; and

provide, for transmission, a prioritized listing of images responsive to a user search query, the prioritized listing of images including a different listing of images for each cluster of compositional similarity based on the metadata of each image associated with the cluster.

14. The system of claim 13, wherein the instructions further cause the one or more processors to:

define a distance function that determines a cosine angle difference between at least two compositional vectors,

wherein each image is processed through the clustering algorithm using the defined distance function.

15. The system of claim 13, wherein the instructions further cause the one or more processors to:

define a distance function that determines a cosine similarity between the compositional vector representing the image and a compositional vector representing one of a plurality of predetermined compositional classes,

wherein each image is processed through the clustering algorithm using the defined distance function, and

wherein the clustering algorithm maps each image to a cluster representing one of the plurality of predetermined compositional classes.

16. The system of claim 15, wherein the distance function determines a cosine angle difference between the compositional vector representing the image and the compositional vector representing the predetermined compositional class.

17. The system of claim 13, wherein the instructions further cause the one or more processors to:

determine a predetermined number of compositional classes;

for each of the predetermined compositional classes, generate an icon representation of a composition corresponding to the compositional class;

provide for display a user-selectable control to filter search results responsive to a user search query, the user-selectable control including one or more icon representations corresponding to respective clusters of compositional classes;

identify a set of images that correspond to a compositional class associated with a user-selected icon representation based on user interaction with the user-selectable control; and

provide, for transmission, a prioritized listing of images from the identified set of images for display on a client device.

18. The system of claim 13, wherein the instructions further cause the one or more processors to:

determine a centroid vector for each of the one or more clusters;

generate an icon representation of a composition corresponding to the centroid vector, the icon representation indicating a compositional class of the cluster;

provide for display a user-selectable control to filter search results responsive to a user search query, the user-selectable control including one or more icon representations corresponding to respective clusters of compositional classes;

identify a set of images that correspond to a compositional class associated with a user-selected icon representation based on user interaction with the user-selectable control; and

provide, for transmission, a prioritized listing of images from the identified set of images for display on a client device.

19. The system of claim 13, wherein the instructions further cause the one or more processors to:

obtain behavioral data from session logs associated with a user;

generate a compositional profile for the user, the compositional profile including an N-dimensional vector where each element of the vector indicates a probability that a next image corresponds to a compositional class associated with the element, where N corresponds to a number of clusters produced; and

for each image in the prioritized listing of images, apply a score based on an element from the N-dimensional vector in the compositional profile, the element corresponding to a compositional class of the image,

wherein images from the prioritized listing of the images are prioritized based on the score applied to each image, and

wherein each image processed through the trained computer-operated convolutional neural network is provided from the obtained behavioral data.

20. A computer-implemented method, comprising:

receiving user input via an application on a client device to initiate an image search, the user input indicating one or more queries that define a specific composition for an image;

generating, in response to the received user input, an image search query from the received user input;

providing for transmission the image search query over a connection to a server, the server including an image search service that obtains a set of images responsive to the image search query based on a cosine similarity between a compositional vector associated with the image search query and one or more compositional vectors of corresponding images from an image collection, the image search service clustering the set of images based on composition similarity using a clustering algorithm associated with a defined distance function; and

receiving a set of search results responsive to the image search query from the server, the set of search results including a prioritized listing of images clustered by composition.