IMAGE PROCESSING APPARATUS, CONTROL METHOD, AND RECORDING MEDIUM

• US 20150286899A1
• Filed: 04/01/2015
• Published: 10/08/2015
• Est. Priority Date: 04/04/2014
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Nakayama, Fumitaka
• Granted Patent: US 9,665,939 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

An image processing apparatus obtains an input image acquired by image capturing, and classifies the input image into a main subject region and a background region. The image processing apparatus determines virtual image capturing conditions corresponding to an output image to be generated, based on the positional relationship between a main subject and a background subject in an image capturing environment when the input image was captured. The image processing apparatus deforms, of the input image, an image classified into the background region, and generates a background image of the output image that corresponds to the determined virtual image capturing conditions. The image processing apparatus composes, of the input image, an image classified into the main subject region and the background image, thereby generating the output image.

Claims

1. An image processing apparatus comprising:

an obtaining unit configured to obtain an input image acquired by image capturing;

a classification unit configured to classify the input image obtained by said obtaining unit into a main subject region and a background region;

a determination unit configured to determine virtual image capturing conditions corresponding to an output image to be generated, based on a positional relationship between a main subject and a subject of the background region in an image capturing environment when the input image was captured;

a generation unit configured to deform, of the input image, an image classified into the background region, and generate a background image of the output image that corresponds to the virtual image capturing conditions determined by said determination unit; and

a composition unit configured to compose, of the input image, an image classified into the main subject region, and the background image generated by said generation unit, thereby generating the output image.

2. The apparatus according to claim 1, wherein

said determination unit includes an information designation unit configured to designate information of at least one of an image capturing position, a focal distance, and an enlargement ratio of a predetermined background subject, which correspond to the output image, and

said determination unit determines the image capturing position and the focal distance which correspond to the output image, as the virtual image capturing conditions based on the information designated by the information designation unit, and the positional relationship in the image capturing environment.

3. The apparatus according to claim 2, wherein said generation unit generates the background image by coordinate transformation based on the focal distance, and a moving amount of an image capturing position corresponding to the output image from an image capturing position corresponding to the input image.

4. The apparatus according to claim 1, wherein

said classification unit the input image into the main subject region and the background region based on information corresponding to distances of the subjects in the image capturing environment on the depth direction.

5. The apparatus according to claim 1, wherein

said generation unit generates the background image by deforming the image classified into the background region based on distance information of the background region.

6. The apparatus according to claim 1, wherein

said classification unit performs the classification for the main subject region and the back ground region based on a distribution of defocus amounts obtained from the correlation relationship between a plurality of images.

7. The apparatus according to claim 1, wherein

said classification unit further classifies the image classified into the background region into a plurality of regions of different depth ranges based on the positional relationship in the image capturing environment, and

said generation unit performs different deformations on images classified into the respective regions.

8. The apparatus according to claim 1, wherein

said composition unit composes the image classified into the main subject region to make a position and size of the image in the output image be equal to a position and size of the image in the input image, and

said determination unit determines the virtual image capturing conditions based on conditions of the position and size of the image classified into the main subject region in the output image.

9. The apparatus according to claim 1, wherein

said composition unit includes an enlargement ratio designation unit configured to designate a size of the image classified into the main subject region,

said determination unit determines the virtual image capturing conditions based on the size of the image classified into the main subject region that is designated by the enlargement ratio designation unit, and

said composition unit deforms the image classified into the main subject region to have the size designated by the enlargement ratio designation unit, and composes the deformed image with the background image.

10. The apparatus according to claim 1, wherein said composition unit applies a low-pass filter to the background image based on the positional relationship in the image capturing environment, and composes the background image.

11. The apparatus according to claim 1, wherein said composition unit performs weighted addition composition of the image classified into the main subject region and the background image in a peripheral region of a boundary of the main subject region.

13. A computer-readable recording medium recording a program for causing a computer to function as each unit of an image processing apparatus defined in claim 1.

12. A control method of an image processing apparatus comprising:

an obtaining step of obtaining an input image acquired by image capturing;

a classification step of classifying the input image obtained in the obtaining step into a main subject region and a background region;

a determination step of determining virtual image capturing conditions corresponding to an output image to be generated, based on a positional relationship between a main subject and a subject of the background region in an image capturing environment when the input image was captured;

a generation step of deforming, of the input image, an image classified into the background region, and generating a background image of the output image that corresponds to the virtual image capturing conditions determined in the determination step; and

a composition step of composing, of the input image, an image classified into the main subject region, and the background image generated in the generation step, thereby generating the output image.