SYSTEMS, METHODS, AND COMPUTER PROGRAM PRODUCTS FOR SEARCHING AND SORTING IMAGES BY AESTHETIC QUALITY

• US 20160098844A1
• Filed: 05/29/2015
• Published: 04/07/2016
• Est. Priority Date: 10/03/2014
• Status: Active Grant

• Current Assignee: EyeEm Mobile GmbH
• Original Assignee: EyeEm Mobile GmbH
• Inventors: SHAJI, Appu, RIZK, Ramzi
• Granted Patent: US 9,659,384 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/002 using neural networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

A system, method, and computer program product for assigning an aesthetic score to an image. A method of the present invention includes receiving an image. The method further includes executing a neural network on the image to generate learned features. The method further includes applying a machine-learned model to assign an aesthetic score to the image, where a more aesthetically-pleasing image is given a higher aesthetic score and a less aesthetically-pleasing image is given a lower aesthetic score. The learned features are inputs to the machine-learned model.

Claims

1. A method for assigning an aesthetic score to an image, comprising:

receiving an image;

executing a neural network on the image to generate learned features; and

applying a machine-learned model to assign an aesthetic score to the image, wherein a more aesthetically-pleasing image is given a higher aesthetic score and a less aesthetically-pleasing image is given a lower aesthetic score,

wherein the learned features are inputs to the machine-learned model.

2. The method of claim 1, wherein the neural network is a deep neural network and the learned features are deep-learnt features.

3. The method of claim 2, wherein the neural network includes a stack of filters followed by one or more fully connected layers, wherein the filters include at least one of convolution filters, rectified-linear unit filters, and max-pooling filters.

4. The method of claim 3, wherein the learned features includes outputs of one or more layers of the neural network.

5. The method of claim 2, wherein the neural network includes a recursive-neural-network structure.

6. The method of claim 3, further comprising:

extracting manually selected features from the image, wherein the manually selected features are indicative of aesthetic quality; and

encoding the extracted manually selected features into a high-dimensional feature vector,

wherein the neural network includes a fully connected layer which takes the encoded manually selected feature vector as an additional input.

7. The method of claim 6, wherein extracting the manually selected features comprises:

determining a plurality of scales;

for each of the plurality of scales, segmenting the image into a plurality of regions; and

for each of the plurality of scales and the plurality of regions;

computing a plurality of nearest and farthest regions according to a distance function;

computing a local histogram of oriented gradients; and

computing a color.

8. The method of claim 3, further comprising:

extracting manually selected features from the image, wherein the manually selected features are indicative of aesthetic quality;

encoding the extracted manually selected features into a high-dimensional feature vector;

reducing the dimension of the high-dimensional feature vector; and

concatenating the manually selected dimensionally-reduced feature vector and the learned features;

wherein the concatenated features are inputs to the machine-learned model.

9. The method of claim 8, wherein extracting the manually selected features comprises:

determining a plurality of scales;

for each of the plurality of scales, segmenting the image into a plurality of regions; and

for each of the plurality of scales and the plurality of regions;

computing a plurality of nearest and farthest regions according to a distance function;

computing a local histogram of oriented gradients; and

computing a color.

10. The method of claim 1, further comprising:

pre-processing the image for input into the neural network,

wherein the executing the neural network on the image comprises executing the neural network on the pre-processed image.

11. The method of claim 1, wherein the neural network is trained by a process comprising:

sampling a neural-net training set having a plurality of training images, wherein the sampling includes forming triplets (Im₁⁺, Im₂⁺, Im₃⁻

) where Im₁⁺ and Im₂⁺ are members of the neural-net training set that are annotated as positive aesthetic images and Im₃⁻

is a member of the neural-net training set that is annotated as a negative aesthetic image; and

minimizing a loss function, whereby an embedding space is found where positive aesthetic images are close together and negative aesthetic images are spread apart.

12. The method of claim 11, wherein the loss function is a triple hinge loss function max(0, c−

δ

(φ

(Im₁⁺),φ

(Im₂⁺))+δ

(φ

(Im₁⁺),φ

(Im₃⁻

))), where c is a constant, δ

is a distance function in the embedding space, and φ

represents the learned features.

13. The method of claim 11, wherein the loss function is a contrastive loss function.

14. The method of claim 1, wherein the neural network is adapted to a genre or a user from a base neural network by the training process comprising:

providing an adapted training set, wherein the adapted training set includes images that the base neural network was not trained on; and

initializing training parameters based on training parameters from the base neural network.

15. The method of claim 14, wherein the adapted training set includes images that are annotated based on a user.

16. The method of claim 15, wherein the annotation based on a user comprises annotating an image as aesthetically pleasing or not aesthetically pleasing based on whether the user has done at least one of:

positively ranked or liked the image, viewed the image for an amount of time exceeding a predetermined threshold, visited the image for an amount of visits exceeding a predetermined threshold, purchased the image, positively ranked or liked a category associated with the image, positively ranked or liked a photographer associated with the image; and

commented on the image.

17. The method of claim 16, wherein the annotation based on a user further comprises annotating an image based on a similarity of the image to a previously annotated image.

18. The method of claim 15, wherein the annotation based on a user comprises annotating an image as aesthetically pleasing or not aesthetically pleasing based on an analysis of a social network of the user.

19. The method of claim 15, wherein the annotation based on a user comprises annotating an image as aesthetically pleasing or not aesthetically pleasing based on results from a recommendation engine, wherein the recommendation engine is based on user-metadata, image-metadata, and a social network of the user.

20. A method for ranking images by aesthetic quality for display, comprising:

receiving a plurality of images;

assigning an aesthetic score to each of the plurality of images, wherein the aesthetic score is generated by the method of claim 1; and

displaying the plurality of images in an order based on the aesthetic score.

21. The method of claim 20, wherein the displaying the plurality of images comprises displaying a subset of the plurality of images.

22. The method of claim 21, wherein the subset of the plurality of images is limited to a fixed number of images.

23. The method of claim 21, wherein the subset of the plurality of images is based on a minimum aesthetic score.

24. A method for searching images by aesthetic quality, comprising:

receiving a search query;

searching a plurality of images based on the search query, whereby a list of search results is generated; and

displaying the list of search results by the method of claim 20.

25. The method of claim 24, wherein the searching a plurality of images comprises searching a plurality of metadata associated with the plurality of images.

26. A device for assigning an aesthetic score to an image, comprising:

a processor;

a memory coupled to the processor; and

a network interface coupled to the processor,

wherein the processor is configured to;

receive an image;

execute a neural network on the image to generate learned features; and

apply a machine-learned model to assign an aesthetic score to the image, wherein a more aesthetically-pleasing image is given a higher aesthetic score and a less aesthetically-pleasing image is given a lower aesthetic score,

wherein the learned features are inputs to the machine-learned model.

27. A computer program product for assigning an aesthetic score to an image, said computer program product comprising a non-transitory computer readable medium storing computer readable program code embodied in the medium, said computer program product comprising:

program code for receiving an image;

program code for executing a neural network on the image to generate learned features; and

program code for applying a machine-learned model to assign an aesthetic score to the image, wherein a more aesthetically-pleasing image is given a higher aesthetic score and a less aesthetically-pleasing image is given a lower aesthetic score,

wherein the learned features are inputs to the machine-learned model.

28. A method for generating content associated with a topic, the method comprising:

providing a topic;

receiving a set of images associated with the topic;

for each image in the set of images, assigning a score to the image, wherein the score is based on user interactions with the image and an aesthetic score of the image; and

filtering the set of images based on the assigned scores to create a subset of images.

29. The method of claim 28, further comprising creating a static webpage including the subset of images, whereby the static webpage is indexable by search engines.

30. The method of claim 28, wherein filtering the set of images based on the assigned scores to create a subset of images includes removing all but a top K number of the images, wherein K is a natural number.

31. The method of claim 28, wherein filtering the set of images based on the assigned scores to create a subset of images includes removing all images where the assigned score less than a threshold value.

32. The method of claim 30, further comprising:

removing an image from the subset of images based on an input from a user;

replacing the removed image in the subset of images by an image with a next highest score from the set of images.

33. The method of claim 28, wherein the received set of images includes search results for a pre-defined search parameter associated with the topic.

34. The method of claim 28, wherein the received set of images includes images having metadata associated with the topic.

35. The method of claim 34, wherein the metadata includes tags associated with the topic.

36. The method of claim 28, wherein the aesthetic score is based on an aesthetic score adapted to a user.

37. The method of claim 28, wherein the aesthetic score is based on an aesthetic score adapted to a genre.

38. A method for assessing the efficacy of changes to a social network including a plurality of users capable of uploading images to the social network, the method comprising:

determining a first metric for user and content quality for a first time period;

making a change to the social network, wherein the change occurs after the first time period;

after making the change, determining a second metric for user and content quality for a second time period; and

assessing the efficacy of the change based on the determined first and second metrics for user and content quality.

39. The method according to claim 38, further comprising:

receiving a first set of images during the first time period; and

receiving a second set of images during the second time period,

wherein the first metric includes determining a number of good images in the first set, wherein an image is determined to be good based on an aesthetic score associated with the image, and

wherein the second metric includes determining a number of good images in the second set, wherein an image is determined to be good based on an aesthetic score associated with the image.

40. The method according to claim 39, wherein determining an image to be good based on the aesthetic score associated with the image includes using a pre-determined aesthetic score threshold.

41. The method according to claim 39, wherein determining an image to be good based on the aesthetic score associated with the image includes using an adaptively determined aesthetic score threshold.

42. The method of claim 39, wherein the first metric further includes determining a histogram of aesthetic scores associated with the first set of images and wherein the second metric further includes determining a histogram of aesthetic scores associated with the second set of images.

43. The method of claim 38, further comprising:

receiving a first set of users during the first time period; and

receiving a second set of users during the second time period,

wherein the first metric includes determining a number of good users in the first set, wherein a user is determined to be good based on a user EyeRank score, and

wherein the second metric includes determining a number of good users in the second set, wherein a user is determined to be good based on an user EyeRank score associated with the image.

44. The method of claim 43, wherein the first metric further includes determining a distribution of good users in the first set and wherein the second metric further includes determining a distribution of good users in the second set.

45. The method of claim 38, wherein making a change to the social network includes changing a user signup process.

46. The method of claim 38, wherein making a change to the social network includes changing how a user is exposed to content.

47. The method of claim 38, wherein making a change to the social network includes changing a user experience design or user interface design.

48. The method of claim 38, wherein assessing the efficacy of the change includes split A/B testing.