Image processing apparatus, control method, and recording medium

• US 10,304,161 B2
• Filed: 04/26/2017
• Issued: 05/28/2019
• Est. Priority Date: 04/04/2014
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Nakayama, Fumitaka
• Primary Examiner(s): Shah, Utpal D
• Application Number:
  US15/497,807
• Publication Number:
  US 20170228853A1
• Time in Patent Office: 762 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

An image processing apparatus obtains an input image acquired by image capturing, and classifies the input image into a main subject region and a background region. The image processing apparatus determines virtual image capturing conditions corresponding to an output image to be generated, based on the positional relationship between a main subject and a background subject in an image capturing environment when the input image was captured. The image processing apparatus deforms, of the input image, an image classified into the background region, and generates a background image of the output image that corresponds to the determined virtual image capturing conditions. The image processing apparatus composes, of the input image, an image classified into the main subject region and the background image, thereby generating the output image.

Claims

1. An image processing apparatus comprising:

a processor; and

a memory storing instructions which, when executed by the processor, causes the processor to perform operations of the following units of the image processing apparatus;

an obtaining unit configured to obtain a main subject image and a background image by image capturing;

a determination unit configured to determine virtual image capturing conditions, which include a focal distance, corresponding to an output image to be generated, based on a positional relationship between a main subject and a subject of the background image;

a generation unit configured to deform the background image, and generate a background image of the output image that corresponds to the virtual image capturing conditions determined by the determination unit; and

a composition unit configured to compose the main subject image and the generated background image generated by the generation unit, thereby generating the output image.

2. The apparatus according to claim 1, wherein

the determination unit includes an information designation unit configured to designate information of at least one of an image capturing position, a focal distance, and an enlargement ratio of a predetermined background subject, which correspond to the output image, and

the determination unit determines the image capturing position and the focal distance which correspond to the output image, as the virtual image capturing conditions based on the information designated by the information designation unit, and the positional relationship in the image capturing environment.

3. The apparatus according to claim 2, wherein the generation unit generates the generated background image by coordinate transformation based on the focal distance and a moving amount of an image capturing position corresponding to the output image from an image capturing position corresponding to the input image.

4. The apparatus according to claim 3, wherein the obtaining unit classifies one input image obtained by image capturing into the main subject image and the background image.

5. The apparatus according to claim 4, wherein

the obtaining unit classifies the input image into the main subject image and the background image based on information corresponding to distances of the subjects in the image capturing environment on a depth direction.

6. The apparatus according to claim 1, wherein

the generation unit generates the generated background image by deforming the background image based on distance information of the background image.

7. The apparatus according to claim 4, wherein

the obtaining unit performs the classification for the main subject image and the back ground image based on a distribution of defocus amounts obtained from a correlation relationship between a plurality of images.

8. The apparatus according to claim 4, wherein

the obtaining unit further classifies the background image of the input image into a plurality of images of different depth ranges based on the positional relationship in the image capturing environment, and

the generation unit performs different deformations on each of the plurality of regions of the background image.

9. The apparatus according to claim 1, wherein

the composition unit composes the main subject image to make a position and size of the image in the output image be equal to a position and size of the image in the input image, and

the determination unit determines the virtual image capturing conditions based on conditions of the position and size of the main subject image.

10. The apparatus according to claim 1, wherein

the composition unit includes an enlargement ratio designation unit configured to designate a size of the main subject image,

the determination unit determines the virtual image capturing conditions based on the size of the main subject image that is designated by the enlargement ratio designation unit, and

the composition unit deforms the main subject image to have the size designated by the enlargement ratio designation unit, and composes the deformed image with the generated background image.

11. The apparatus according to claim 1, wherein the composition unit applies a low-pass filter to the generated background image based on the positional relationship in the image capturing environment, and composes the generated background image.

12. The apparatus according to claim 1, wherein the composition unit performs weighted addition composition of the main subject image and the generated background image in a peripheral region of a boundary of the main subject image.

13. A control method of an image processing apparatus, comprising:

obtaining a main subject image and a background image by image capturing;

determining virtual image capturing conditions corresponding to an output image to be generated, based on a positional relationship between a main subject and a subject of the background image;

deforming the background image and generating a background image of the output image that corresponds to the determined virtual image capturing conditions; and

composing the main subject image and the generated background image, thereby generating the output image.

14. A non-transitory computer-readable recording medium recording a program for causing a computer to perform:

obtaining a main subject image and a background image by image capturing;

determining virtual image capturing conditions corresponding to an output image to be generated, based on a positional relationship between a main subject and a subject of the background image;

deforming the background image and generating a background image of the output image that corresponds to the determined virtual image capturing conditions; and

composing the main subject image and the generated background image, thereby generating the output image.