Image processing device and program

• US 9,432,641 B2
• Filed: 09/03/2012
• Issued: 08/30/2016
• Est. Priority Date: 09/02/2011
• Status: Active Grant

• Current Assignee: Nikon Corporation
• Original Assignee: Nikon Corporation
• Inventors: Nishi, Takeshi
• Primary Examiner(s): LAM, HUNG H
• Application Number:
  US14/342,404
• Publication Number:
  US 20140204246A1
• Time in Patent Office: 1,457 Days
• Field of Search: 348/208.12, 348/221.1, 348/227.1, 348/229.1, 348/362, 348/FOR.198, 348/FOR.206
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
• CPC Class Codes: H04N 9/77 Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase circuits for matrixing H04N9/67

Abstract

An image processing device includes: an identifying unit that identifies an image area where gradations are lost in a first captured image obtained by capturing a subject image; an image texture image generation unit that generates an image texture image by aligning a second captured image, obtained by capturing an image of a subject matching the subject of the first captured image, with the first captured image and extracting an image texture corresponding to the image area having been identified by the identifying unit; and an interpolation unit that executes interpolation for the image area in the first captured image by synthetically incorporating the image texture image having been generated by the image texture image generation unit with the first captured image.

Claims

1. An image processing device, comprising:

an identifying unit, implemented by a processor, that identifies a first image area including a first plurality of pixels or a second image area including a second plurality of pixels, in a first captured image obtained by capturing a subject image, a first luminance value of each of the first plurality of pixels being equal to or less than a first threshold value and a second luminance value of each of the second plurality of pixels each being equal to or greater than a second threshold value;

an image texture image generation unit, implemented by the processor, that generates an image texture image by aligning a second captured image, obtained by capturing an image of a subject matching the subject of the first captured image, with the first captured image and extracting an image texture corresponding to the image area having been identified by the identifying unit;

an interpolation unit, implemented by the processor, that executes interpolation for the image area in the first captured image by synthetically incorporating the image texture image having been generated by the image texture image generation unit with the first captured image; and

a threshold value setting unit, implemented by the processor, that alters at least one of the first threshold value and second threshold value based upon luminance information of the first captured image.

2. An image processing device according to claim 1,

wherein:

the threshold value setting unit;

generates a luminance histogram of the first captured image and divides the luminance histogram into a plurality of groups;

calculates a first difference between an integrated value calculated for a current frame in correspondence to a first group, which includes a lowest grade assuming a frequency value other than 0, among the plurality of groups and an integrated value calculated for a previous frame in correspondence to the group;

calculates a second difference between an integrated value calculated for the current frame in correspondence to a second group, which includes a highest grade assuming a frequency value other than 0, among the plurality of groups and an integrated value calculated for the previous frame in correspondence to the group;

sets a largest value in the first group for the first threshold value if the first difference is equal to or greater than a predetermined value; and

sets a smallest value in the second group for the second threshold value if the second difference is equal to or greater than a predetermined value.

3. An image processing device according claim 1, wherein:

the image texture includes at least image edges.

4. An image processing device according to claim 3, wherein:

the image texture further includes image luminance gradation information.

5. An image processing device according to claim 1, further comprising:

an image capturing unit that captures the first captured image and the second captured image.

6. A non-transitory computer-readable computer program product containing a control program for image processing, the control program comprising:

an instruction for identifying a first image area including a first plurality of pixels or a second image area including a second plurality of pixels, in a first captured image obtained by capturing a subject image, a first luminance value of each of the first plurality of pixels being equal to or less than a first threshold value and a second luminance value of each of the second plurality of pixels being equal to or great than a second threshold value;

an instruction for generating an image texture image by aligning a second captured image obtained by capturing an image of a subject matching the subject of the first captured image with the first captured image, and by extracting an image texture corresponding to the image area having been identified;

an instruction for executing interpolation for the image area in the first captured image by synthetically incorporating the image texture image having been generated and the first captured image; and

an instruction for altering at least one of the first threshold value and the second threshold value based upon luminance information of the first captured image.

7. An image processing device, comprising:

an image frame extracting unit, implemented by the processor, that extracts a target image frame subject to exposure correction, among image frames constituting a video image recorded in a recording medium;

an identifying unit, implemented by the processor, that identifies a first image area including a first plurality of pixels or a second image area including a second plurality of pixels, in the target image frame, a first luminance value of each of the first plurality of pixels being equal to or less than a first threshold value and a second luminance value of each of the second plurality of pixels being equal to or greater than a second threshold value;

an estimating unit, implemented by the processor, that estimates a texture of the first image area of the second image area by using an image frame, other than the target image frame, constituting the video image;

an interpolation unit, implemented by the processor, that executes interpolation for the first image area or the second image area in the image frame based upon estimation results provided by the estimating unit; and

a threshold value setting unit, implemented by the processor, that alters at least one of the first threshold value and the second threshold value based upon luminance information of the target image frame.

8. An image processing device according to claim 7, wherein:

the identifying unit identifies the first image area or the second image area based upon luminance distribution within the image frame.

9. An image processing device according to claim 7, wherein:

the estimating unit generates a texture image for the first image area or the second image area by using an image frame other than the target image frame; and

the interpolation unit executes interpolation by incorporating the texture image in the first image area the second image area.

10. An image processing device according to claim 7, further comprising:

an image sensor that captures an image formed with subject light, wherein;

a video image generated based upon image capturing signals provided from the image sensor is recorded in the recording medium.