IMAGE PICKUP APPARATUS, IMAGE PICKUP METHOD, PROGRAM, AND IMAGE PROCESSING APPARATUS

• US 20170318238A1
• Filed: 11/26/2015
• Published: 11/02/2017
• Est. Priority Date: 12/10/2014
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: SUGIYAMA, Toshinobu
• Granted Patent: US 10,462,390 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: H01L 27/14621 Colour filter arrangements
• CPC Class Codes: H01L 27/14649 Infrared imagers
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 25/50 Control of the SSIS exposure
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors

Abstract

The present disclosure relates to an image pickup apparatus, an image pickup method, a program, and an image processing apparatus that enable a color image to be generated on the basis of an infrared image and a visible image captured using an image pickup device that uses a focal plane reading system.

An image pickup apparatus according to the present disclosure includes: an image pickup device that generates, in a one frame period, sub-frame images in a number corresponding to 3 or more sub-frame periods; an infrared light irradiation unit that turns on/off irradiation of infrared light onto an image pickup range in a time length unit that is the same as the sub-frame period in the one frame period; and a color image generation unit that generates a color image in a predetermined frame rate on the basis of an infrared image based on the sub-frame image in which a period during which the infrared light is irradiated is included in an exposure time and a visible image based on the sub-frame image in which the period during which the infrared light is irradiated is not included in the exposure time. The present disclosure is applicable to a surveillance camera, for example.

Claims

1. An image pickup apparatus, comprising:

an image pickup device that divides one frame period corresponding to a predetermined frame rate into 3 or more sub-frame periods and sweeps pixel values of an entire screen for each of the sub-frame periods using a focal plane reading system to generate, in the one frame period, sub-frame images in a number corresponding to the number of sub-frame periods;

an infrared light irradiation unit that turns on/off irradiation of infrared light onto an image pickup range in a time length unit that is the same as the sub-frame period in the one frame period; and

a color image generation unit that generates a color image at the predetermined frame rate on the basis of an infrared image based on the sub-frame image in which a period during which the infrared light is irradiated is included in an exposure time and a visible image based on the sub-frame image in which the period during which the infrared light is irradiated is not included in the exposure time.

2. The image pickup apparatus according to claim 1, wherein

the color image generation unit includes

an infrared image generation unit that generates an infrared image from the plurality of sub-frame images in which the period during which the infrared light is irradiated is included in the exposure time.

3. The image pickup apparatus according to claim 2, wherein

the infrared image generation unit generates the infrared image from the plurality of sub-frame images whose luminance gradations are in a complementary relationship, the plurality of sub-frame images having the period during which the infrared light is irradiated included in the exposure time.

4. The image pickup apparatus according to claim 2, wherein

the color image generation unit further includes

a visible image generation unit that generates the visible image from the plurality of sub-frame images in which the period during which the infrared light is irradiated is not included in the exposure time.

5. The image pickup apparatus according to claim 2, wherein

the color image generation unit further includes

an infrared image conversion unit that extracts luminance information from the infrared image,

a visible image conversion unit that extracts color difference information from the visible image, and

a color image conversion unit that converts the luminance information of the infrared image and the color difference information of the visible image into color information of the color image.

6. The image pickup apparatus according to claim 5, wherein

the visible image conversion unit also extracts luminance information from the visible image, and

the color image conversion unit converts the luminance information of the infrared image and the color difference information of the visible image into the color information of the color image, the luminance information and the color difference information having been corrected using the luminance information of the visible image.

7. The image pickup apparatus according to claim 2, wherein

the color image generation unit further includes

a first 3DNR unit that carries out 3DNR processing including time direction processing on the infrared image, and

a second 3DNR unit that carries out the 3DNR processing including the time direction processing on the visible image, and

a feedback ratio of the time direction processing in the first 3DNR unit and a feedback ratio of the time direction processing in the second 3DNR unit differ.

8. The image pickup apparatus according to claim 7, wherein

the second 3DNR unit carries out the 3DNR processing on the visible image using a moving subject detection result supplied from the first 3DNR unit.

9. The image pickup apparatus according to claim 8, wherein

the color image generation unit further includes

a 2DNR unit that carries out 2DNR processing on the visible image using an edge detection result supplied from the first 3DNR unit.

10. The image pickup apparatus according to claim 2, wherein

a light reception surface of the image pickup device is covered by a color filter in a Bayer array, an RGB-IR filter, or an RGB-W filter.

11. The image pickup apparatus according to claim 2, wherein

the image pickup device includes an electronic shutter function, and the exposure time during image pickup of the sub-frame images that are to become a base of the infrared image differs in a time length from the exposure time during image pickup of the sub-frame images that are to become a base of the visible image.

12. The image pickup apparatus according to claim 2, wherein

the image pickup device includes an amplification unit that amplifies pixel signals and an electronic shutter function, and a gain of the amplification unit differs between the time of image pickup of the sub-frame images that are to become a base of the infrared image and the time of image pickup of the sub-frame images that are to become a base of the visible image.

13. An image pickup method for an image pickup apparatus including an image pickup device, an infrared light irradiation unit, and a color image generation unit, the method comprising the steps of:

dividing, by the image pickup device, one frame period corresponding to a predetermined frame rate into 3 or more sub-frame periods and sweeping pixel values of an entire screen for each of the sub-frame periods using a focal plane reading system to generate, in the one frame period, sub-frame images in a number corresponding to the number of sub-frame periods;

turning on/off, by the infrared light irradiation unit, irradiation of infrared light onto an image pickup range in a time length unit that is the same as the sub-frame period in the one frame period; and

generating, by the color image generation unit, a color image at the predetermined frame rate on the basis of an infrared image based on the sub-frame image in which a period during which the infrared light is irradiated is included in an exposure time and a visible image based on the sub-frame image in which the period during which the infrared light is irradiated is not included in the exposure time.

14. A program for controlling a computer of an image pickup apparatus including an image pickup device and an infrared light irradiation unit to execute processing including the steps of:

dividing, by the image pickup device, one frame period corresponding to a predetermined frame rate into 3 or more sub-frame periods and sweeping pixel values of an entire screen for each of the sub-frame periods using a focal plane reading system to generate, in the one frame period, sub-frame images in a number corresponding to the number of sub-frame periods;

turning on/off, by the infrared light irradiation unit, irradiation of infrared light onto an image pickup range in a time length unit that is the same as the sub-frame period in the one frame period; and

generating a color image at the predetermined frame rate on the basis of an infrared image based on the sub-frame image in which a period during which the infrared light is irradiated is included in an exposure time and a visible image based on the sub-frame image in which the period during which the infrared light is irradiated is not included in the exposure time.

15. An image processing apparatus that receives, from an image pickup apparatus including an image pickup device that divides one frame period corresponding to a predetermined frame rate into 3 or more sub-frame periods and sweeps pixel values of an entire screen for each of the sub-frame periods using a focal plane reading system to generate, in the one frame period, sub-frame images in a number corresponding to the number of sub-frame periods, and an infrared light irradiation unit that turns on/off irradiation of infrared light onto an image pickup range in a time length unit that is the same as the sub-frame period in the one frame period, an input of the sub-frame images captured by the image pickup apparatus, the image processing apparatus comprising

a color image generation unit that generates a color image at the predetermined frame rate on the basis of an infrared image based on the sub-frame image in which a period during which the infrared light is irradiated is included in an exposure time and a visible image based on the sub-frame image in which the period during which the infrared light is irradiated is not included in the exposure time.