Machine learning to predict cognitive image composition

• US 10,592,743 B2
• Filed: 08/24/2017
• Issued: 03/17/2020
• Est. Priority Date: 08/24/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Borrel, Paul, Buoro, Alvaro B., Barros, Ruberth A. A., Salles Chevitarese, Daniel
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Mangialaschi, Tracy
• Application Number:
  US15/685,016
• Publication Number:
  US 20190065848A1
• Time in Patent Office: 936 Days
• Field of Search: None
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

An automatic method of determining an image composition procedure that generates a new image visualization based on aggregations and variations of input images. A set of input images is received. Visual features are extracted from the input images. Context associated with input images is received. Based on the extracted visual features and the context associated with the input images, a composition procedure comprising a set of image operations to apply on the set of input images is learned. One or more image operations in the composition procedure are determined to present to a user. A difference visualization image associated with the input images may be generated by executing the one or more image operations.

Claims

1. An automatic method of determining an image composition procedure that generates a new image visualization based on aggregations and variations of input images, the method executed by a hardware processor, comprising:

receiving a set of input images;

extracting visual features from the input images;

receiving context associated with the input images;

learning based on the extracted visual features and the context associated with the input images, a composition procedure comprising at least a sequence of set of image operations to apply on the set of input images, the sequence of set of image operations obeying a sequential structure, wherein a previous operation in the sequence creates data for a next operation in the sequence, the sequence of set of image operations including at least a pre-processing operation, a quantification operation, a visualization object creation operation, and an objects visualization operation;

determining image operations in the composition procedure to present to a user based on the learning; and

generating a difference visualization image associated with the input images by executing the image operations, the difference visualization image representing an aggregation of the set of input images.

2. The method of claim 1, further comprising:

monitoring execution of image operations on a training set of images, the image operations generating a difference visualization image associated with the training set of images, wherein multiple executions on multiple sets of images are monitored that generate corresponding multiple difference visualization images; and

storing in a knowledgebase, the image operations, visual features in the training set of images, and context associated with the training set of images.

3. The method of claim 2, wherein the learning is further based on the knowledgebase.

4. The method of claim 1, further comprising receiving from the user a rating associated with a suggested image operation, and the learning is further based on the rating.

5. The method of claim 4, wherein the rating is used by a reinforcement learning process to improve a reward associated to actions, wherein additional composition procedure comprising a sequence of the actions is presented to the user.

6. The method of claim 1, wherein the context comprises whether the input images are of synthetic objects or natural objects, whether the input images are 2-dimensional or 3-dimensional, image count in the input images, a degree of variability among the input images, a number of colors in the input images, whether a legend is specified in the input images, a domain corresponding to the input images, and an objective in analyzing the input images.

7. The method of claim 1, wherein the step of extracting visual features from the input images comprises executing a Deep Convolutional Activation Feature (DeCAF) extractor to learn visual features from the set of input images.

8. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:

receive a set of input images;

extract visual features from the input images;

receive context associated with the input images;

learn based on the extracted visual features and the context associated with the input images, a composition procedure comprising at least a sequence of set of image operations to apply on the set of input images, the sequence of set of image operations obeying a sequential structure, wherein a previous operation in the sequence creates data for a next operation in the sequence, the sequence of set of image operations including at least a pre-processing operation, a quantification operation, a visualization object creation operation, and an objects visualization operation;

determine image operations in the composition procedure to present to a user based on the learning; and

generate a difference visualization image associated with the input images by executing the image operations, the difference visualization image representing an aggregation of the set of input images.

9. The computer program product of claim 8, wherein the device is further caused to:

monitor execution of image operations on a training set of images, the image operations generating a difference visualization image associated with the training set of images, wherein multiple executions on multiple sets of images are monitored that generate corresponding multiple difference visualization images; and

store in a knowledgebase, the image operations, visual features in the training set of images, and context associated with the training set of images.

10. The computer program product of claim 9, wherein the learning is further based on the knowledgebase.

11. The computer program product of claim 8, wherein the device is further caused to receive from the user a rating associated with a suggested image operation, and the learning is further based on the rating.

12. The computer program product of claim 11, wherein the rating is used by a reinforcement learning process to improve a reward associated to actions, wherein additional composition procedure comprising a sequence of the actions is presented to the user.

13. The computer program product of claim 8, wherein the context comprises whether the input images are of synthetic objects or natural objects, whether the input images are 2-dimensional or 3-dimensional, image count in the input images, a degree of variability among the input images, a number of colors in the input images, whether a legend is specified in the input images, a domain corresponding to the input images, and an objective in analyzing the input images.

14. The computer program product of claim 8, wherein to extract visual features from the input images, the device is further caused to execute a Deep Convolutional Activation Feature (DeCAF) extractor to learn visual features from the set of input images.

15. An automatic system of determining an image composition procedure that generates a new image visualization based on aggregations and variations of input images, comprising:

at least one hardware processor;

a display device couple to the at least one hardware processor;

the at least one hardware processor operable to receiving a set of input images;

the at least one hardware processor operable to extract visual features from the input images;

the at least one hardware processor operable to receive context associated with the input images;

the at least one hardware processor operable to learn based on the extracted visual features and the context associated with the input images, a composition procedure comprising at least a sequence of set of image operations to apply on the set of input images, the sequence of set of image operations obeying a sequential structure, wherein a previous operation in the sequence creates data for a next operation in the sequence, the sequence of set of image operations including at least a pre-processing operation, a quantification operation, a visualization object creation operation, and an objects visualization operation;

the at least one hardware processor operable to determine based on the learning, image operations in the composition procedure to present to a user on the display device; and

the at least one hardware processor operable to generate a difference visualization image associated with the input images by executing the image operations, the difference visualization image representing an aggregation of the set of input images.

16. The system of claim 15, further comprising:

a knowledgebase coupled to the at least one hardware processor, wherein the at least one hardware processor is further operable to monitor execution of image operations on a training set of images, the image operations generating a difference visualization image associated with the training set of images, wherein multiple executions on multiple sets of images are monitored that generate corresponding multiple difference visualization images, and to store in the knowledgebase, the image operations, visual features in the training set of images, and context associated with the training set of images.

17. The system of claim 16, wherein the at least one hardware processor learns further based on the knowledgebase.

18. The system of claim 15, wherein the at least one hardware processor operable to receive from the user a rating associated with a suggested image operation, and the at least one hardware processor learns further based on the rating.

19. The system of claim 18, wherein the rating is used by a reinforcement learning process to improve a reward associated with actions, wherein additional composition procedure comprising a sequence of the actions is presented to the user.

20. The system of claim 15, wherein the context comprises whether the input images are of synthetic objects or natural objects, whether the input images are 2-dimensional or 3-dimensional, image count in the input images, a degree of variability among the input images, a number of colors in the input images, whether a legend is specified in the input images, a domain corresponding to the input images, and an objective in analyzing the input images.