Image processing apparatus, image processing apparatus control method, image pickup apparatus, and image pickup apparatus control method

• US 10,455,149 B2
• Filed: 01/16/2018
• Issued: 10/22/2019
• Est. Priority Date: 08/20/2014
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Nakayama, Fumitaka
• Primary Examiner(s): Haskins, Twyler L
• Assistant Examiner(s): Tejano, Dwight Alex C
• Application Number:
  US15/872,465
• Publication Number:
  US 20180146135A1
• Time in Patent Office: 644 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 23/672 based on the phase difference signals
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/702 SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout
• CPC Class Codes: H04N 25/704 Pixels specially adapted for focusing, e.g. phase difference pixel sets
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

An image processing apparatus including an acquiring unit configured to acquire an image signal of a plurality of images obtained by picking up an object image that is formed by a photographing optical system; a determining unit configured to determine a defocus amount of the object image by using the image signal of the plurality of images; and a conversion unit configured to perform, on the defocus amount, gradation conversion that has at least one conversion characteristic out of a plurality of different conversion characteristics, and to output information that is based on the defocus amount converted by the gradation conversion.

Claims

1. An image processing apparatus comprising:

one or more processors; and

a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as;

an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and

a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion,

wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that more gradations are allocated to distance information near an in-focus region as compared with distance information not near the in-focus region,

wherein the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from a plurality of conversion characteristics, and

in case where an application for which distance resolution near the in-focus region is necessary is a destination of the output, the conversion unit is configured to determine the conversion characteristic as a conversion characteristic that more resolutions are allocated to the distance information near the in-focus region as compared with the distance information not near the in-focus region.

2. The image processing apparatus according to claim 1, wherein

the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and

there is further provided a conversion characteristic that the gradations are asymmetrically allocated to distance information in front of and behind distance information of a focused object.

3. The image processing apparatus according to claim 1, wherein

the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and

there is further provided a conversion characteristic that the resolution conversion varies depending on an F-value of a lens of the photographing optical system.

4. The image processing apparatus according to claim 1, wherein

the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and

there is further provided a conversion characteristic that, in accordance with an F-value of a lens of the photographing optical system, as the F-value becomes larger, the gradations to be allocated to larger distance information are made fewer.

5. The image processing apparatus according to claim 1, wherein

the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and

there is further provided a conversion characteristic that the resolution conversion varies depending on a base length in the pickup of the object image.

6. The image processing apparatus according to claim 1, wherein

the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and

there is further provided a conversion characteristic that the resolution conversion varies depending on an ISO sensitivity in the pickup of the object image.

7. The image processing apparatus according to claim 1, wherein

the conversion unit is configured to perform the resolution conversion by selectively determining the conversion characteristic from the plurality of conversion characteristics, and

in a case where an application for which a relatively wider distance measurement range is necessary is a destination of the output, the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that liner resolution conversion is to be performed.

8. The image processing apparatus according to claim 1, wherein

the distance information is information of an object distance,

the determining unit further comprises a generating unit configured to generate, as the distance information, the information of the object distance from a defocus amount acquired based on the image signals of the plurality of images, and

the conversion unit is configured to convert the object distance depending on the conversion characteristic.

9. The image processing apparatus according to claim 1, wherein

the conversion unit is configured to determine one or more conversion characteristics respectively for the distance information, convert the distance information depending on the respective conversion characteristics, and output a plurality of results acquired by the conversion.

10. The image processing apparatus according to claim 1, wherein

the image signals of the plurality of images include image signals of a plurality of images of the object image different from one another in blurring, image signals of a plurality of images acquired by picking up the object image from different regions of a pupil of the photographing optical system, or image signals of a plurality of images acquired by picking up an object with a plurality of image pickup units.

11. The image processing apparatus according to claim 1, wherein

the distance information is based on a value indicating a degree of match between the plurality of images or a blurring evaluation value of the plurality of images.

12. The image processing apparatus according to claim 1, wherein

the distance information is information which is calculated from the image signals of the plurality of images and corresponds to an object distance of an object in the image.

13. The image processing apparatus according to claim 1, wherein

distance information converted using the conversion characteristic by the conversion unit is output as an object distance map.

14. The image processing apparatus according to claim 1, wherein

the distance information is information representing a defocus amount calculated from the image signals of the plurality of images.

15. The image processing apparatus according to claim 1, wherein

the distance information converted using the conversion characteristic by the conversion unit is output as a defocus map.

16. The image processing apparatus according to claim 1, further comprising

an image processing unit configured to perform an image process using distance information converted by the conversion unit.

17. The image processing apparatus according to claim 16, wherein

the image process includes image deformation process of deforming, by using distance information converted by the conversion unit, each region of the image depending on distance information.

18. An image processing apparatus comprising:

one or more processors; and

a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as;

an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and

a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion,

wherein the conversion unit configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that gradations are asymmetrically allocated to distance information in front of and behind distance information of a focused object.

19. An image processing apparatus comprising:

one or more processors; and

a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as;

an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and

a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion,

wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that resolution conversion varies depending on an F-value of a lens of the photographing optical system.

20. An image processing apparatus comprising:

one or more processors; and

a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as;

an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and

a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion,

wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that, in accordance with an F-value of a lens of the photographing optical system, as the F-value becomes larger, gradations to be allocated to larger distance information are made fewer.

21. An image processing apparatus comprising:

one or more processors; and

a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as;

an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and

a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion,

wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that resolution conversion varies depending on a base length in the pickup of the object image.

22. An image processing apparatus comprising:

one or more processors; and

a memory storing instructions which, when the instructions are executed by the processor, cause the image processing apparatus to function as;

an acquiring unit configured to acquire image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

a determining unit configured to determine distance information of the object image by using the image signals of the plurality of images; and

a conversion unit configured to perform resolution conversion of converting resolution of the distance information and output a result of the resolution conversion,

wherein the conversion unit is configured to perform the resolution conversion of the distance information and output the result of the resolution conversion, by using a conversion characteristic that resolution conversion varies depending on an ISO sensitivity in the pickup of the object image.

23. An image processing method comprising:

acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

determining distance information of the object image by using the image signals of the plurality of images; and

performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion,

wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that more gradations are allocated to distance information near an in-focus region as compared with distance information not near the in focus region,

wherein performing the resolution conversion includes selectively determining the conversion characteristic from a plurality of conversion characteristics, and

in a case where an application for which distance resolution near the in-focus region is necessary is a destination of the output, determining the conversion characteristic as a conversion characteristic that more resolutions are allocated to the distance information near the in-focus region as compared with the distance information not near the in-focus region.

24. An image processing method comprising:

acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

determining distance information of the object image by using the image signals of the plurality of images; and

performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion,

wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that gradations are asymmetrically allocated to distance information in front of and behind distance information of a focused object.

25. An image processing method comprising:

acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

determining distance information of the object image by using the image signals of the plurality of images; and

performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion,

wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that resolution conversion varies depending on an F-value of a lens of the photographing optical system.

26. An image processing method comprising:

acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

determining distance information of the object image by using the image signals of the plurality of images; and

performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion,

wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that, in accordance with an F-value of a lens of the photographing optical system, as the F-value becomes larger, gradations to be allocated to larger distance information are made fewer.

27. An image processing method comprising:

acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

determining distance information of the object image by using the image signals of the plurality of images; and

performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion,

wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that resolution conversion varies depending on a base length in the pickup of the object image.

28. An image processing method comprising:

acquiring image signals of a plurality of images acquired by picking up an object image formed by a photographing optical system;

determining distance information of the object image by using the image signals of the plurality of images; and

performing resolution conversion of converting resolution of the distance information and outputting a result of the resolution conversion,

wherein the resolution conversion of the distance information is performed and the result of the resolution conversion is output, by using a conversion characteristic that resolution conversion varies depending on an ISO sensitivity in the pickup of the object image.