Image processing apparatus, method, and computer program with pixel brightness-change detection and value correction

• US 8,150,201 B2
• Filed: 03/04/2008
• Issued: 04/03/2012
• Est. Priority Date: 03/09/2007
• Status: Active Grant

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kasai, Masanori, Oike, Yusuke
• Primary Examiner(s): Repko, Jason M
• Assistant Examiner(s): Thirugnanam, Gandhi
• Application Number:
  US12/041,876
• Publication Number:
  US 20080219585A1
• Time in Patent Office: 1,491 Days
• Field of Search: 382/274
• US Class Current: 382/274
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures

Abstract

An image processing apparatus includes the following elements. An image input unit receives a long-time exposure image and a short-time exposure image. An image analysis unit detects a brightness-change pixel in which a brightness change has occurred during a photographic period on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image. A pixel value correction unit corrects a pixel value of the detected brightness-change pixel. In the pixel value correction unit, a combined image generator selectively combines pixel values in the long-time exposure image and pixel values in the short-time exposure image to generate a combined image; an intermediate image generator generates a blurred image of the combined image; and an output image generator determines a pixel value of the detected brightness-change pixel using a pixel value of a corresponding pixel in each of the combined image and the blurred image.

Claims

1. An image processing apparatus comprising:

a processor configured to;

receive input image data comprising a long-time exposure image and a short-time exposure image;

detect a brightness-change pixel in which it is determined that a brightness change has occurred during a photographic period corresponding to a long-time exposure time on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image; and

correct a pixel value of the brightness-change pixel detected by the image analysis unit, wherein the pixel value correction includes;

selectively combining pixel values in the long-time exposure image and pixel values in the short-time exposure image to generate a combined image,

generating an intermediate image, the intermediate image being a blurred image of the combined image, and

determining a pixel value of the brightness-change pixel using a pixel value of a corresponding pixel in the combined image and a pixel value of a corresponding pixel in the intermediate image, the corresponding pixels in the combined and intermediate images being located at positions corresponding to the brightness-change pixel.

2. The image processing apparatus according to claim 1, wherein the processor is configured to determine a pixel value of the corresponding pixel in an output image using the following equation:

[Rv]=a×

[Dv]+b×

[Mv]

where [Rv] denotes the pixel value of the corresponding pixel in the output image, [Dv] denotes the pixel value of the corresponding pixel in the combined image, [Mv] denote the pixel value of the corresponding pixel in the intermediate image, and a and b denote coefficients, wherein a+b=1.

3. The image processing apparatus according to claim 1, wherein the processor is configured to detect a pixel position at which an effective output value is obtained from the short-time exposure image and at which an effective output value is obtained from the long-time exposure image, and is configured to determine that a pixel that is located at the detected pixel position is a brightness-change pixel whose brightness level changes from a high brightness level to a low brightness level.

4. The image processing apparatus according to claim 1, wherein the processor is configured to detect a pixel position at which a pixel value in the long-time exposure image is saturated and at which no effective output value is obtained from the short-time exposure image, and is configured to determine that a pixel that is located at the detected pixel position is a brightness-change pixel whose brightness level changes from a low brightness level to a high brightness level.

5. The image processing apparatus according to claim 1, wherein the processor is configured to perform image conversion on a detection-result image that is based on the detected brightness-change pixel using a smoothing filter, and is configured to exclude a brightness-change pixel that is erroneously detected on the basis of an image obtained as a result of the image conversion.

6. The image processing apparatus according to claim 1, wherein the processor is configured to perform image conversion on a detection-result image that is based on the detected brightness-change pixel using a filter, and is configured to enlarge a brightness-change pixel region.

7. The image processing apparatus according to claim 6, wherein the processor is configured to determine a pixel value of the corresponding pixel in an output image using coefficients by the following equation:

[Rv]=a×

[Dv]+b×

[Mv]

where [Rv] denotes the pixel value of the corresponding pixel in the output image, [Dv] denotes the pixel value of the corresponding pixel in the combined image, [Mv] denotes the pixel value of the corresponding pixel in the intermediate image, and a and b denote the coefficients, wherein a+b=1, and

wherein the processor is further configured to determine the pixel value of the corresponding pixel in the output image by using, as the coefficients, a first set of coefficients used to determine pixel values of a brightness-change pixel region that is detected on the basis of the combined image, and a second set of coefficients to determine pixel values of the brightness-change pixel region enlarged using the filter, the first set of coefficients and the second set of coefficients having different values from each other.

8. The image processing apparatus according to claim 1, wherein the processor is configured to apply a smoothing filter to the combined image to generate a blurred image.

9. The image processing apparatus according to claim 1, wherein the processor is configured to reduce and enlarge the combined image to generate a blurred image.

10. The image processing apparatus according to claim 1, wherein the processor is further configured to reduce a color saturation of the combined image generated to generate a saturation-reduced image, and

wherein the processor is further configured to receive the saturation-reduced image and to generate a blurred image.

11. The image processing apparatus according to claim 1, wherein the processor is further configured to perform an image correction process of applying a low pass filter.

12. The image processing apparatus according to any of claims 1 to 11, wherein an exposure period in which the long-time exposure image is generated includes an exposure period in which the short-time exposure image is generated, and

wherein the short-time exposure image and the long-time exposure image are images generated from an identical pixel by a single imaging element.

13. An imaging apparatus comprising:

an imaging device configured to generate, by a processor, image data comprising a long-time exposure image and a short-time exposure image, wherein the processor is configured to;

detect a brightness-change pixel in which it is determined that a brightness change has occurred during a photographic period corresponding to a long-time exposure time on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image; and

correct a pixel value of the brightness-change pixel detected by the image analysis unit, wherein the correcting a pixel value includes;

selectively combining pixel values in the long-time exposure image and pixel values in the short-time exposure image to generate a combined image,

generating an intermediate image, the intermediate image being a blurred image of the combined image, and

determining a pixel value of the brightness-change pixel detected by the image analysis unit using a pixel value of a corresponding pixel in the combined image and a pixel value of a corresponding pixel in the intermediate image, the corresponding pixel in the combined and intermediate images being located at positions corresponding to the brightness-change pixel.

14. The imaging apparatus according to claim 13, wherein an exposure period in which the long-time exposure image is generated includes an exposure period in which the short-time exposure image is generated, and

wherein the imaging device is configured to generate the short-time exposure image and the long-time exposure image from an identical pixel using a single imaging element.

15. An image processing method for performing image processing in an image processing apparatus, the image processing method comprising the steps of:

by an image input unit, receiving a long-time exposure image and a short-time exposure image;

by an image analysis unit, detecting a brightness-change pixel in which it is determined that a brightness change has occurred during a photographic period corresponding to a long-time exposure time on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image; and

by a pixel value correction unit, correcting a pixel value of the detected brightness-change pixel,

wherein the step of correcting includes the steps of

generating a combined image by selectively combining pixel values in the long-time exposure image and pixel values in the short-time exposure image,

generating an intermediate image, the intermediate image being a blurred image of the combined image, and

determining a pixel value of the brightness-change pixel detected by the image analysis unit using a pixel value of a corresponding pixel in the combined image and a pixel value of the corresponding pixel in the intermediate image, the corresponding pixel in the combined and intermediate images being located at positions corresponding to the brightness-change pixel.

16. The image processing method according to claim 15, wherein the step of determining includes the step of:

determining a pixel value of the corresponding pixel in an output image using the following equation;

[Rv]=a×

[Dv]+b×

[Mv]

where [Rv] denotes the pixel value of the corresponding pixel in the output image, [Dv] denotes the pixel value of the corresponding pixel in the combined image, [Mv] denote the pixel value of the corresponding pixel in the intermediate image, and a and b denote coefficients, wherein a+b=1.

17. The image processing method according to claim 15, wherein the step of detecting includes the step of:

detecting a pixel position at which an effective output value is obtained from the short-time exposure image and at which an effective output value is obtained from the long-time exposure image to determine that a pixel that is located at the detected pixel position is a brightness-change pixel whose brightness level changes from a high brightness level to a low brightness level.

18. The image processing method according to claim 15, wherein the step of detecting includes the step of:

detecting a pixel position at which a pixel value in the long-time exposure image is saturated and at which no effective output value is obtained from the short-time exposure image to determine that a pixel that is located at the detected pixel position is a brightness-change pixel whose brightness level changes from a low brightness level to a high brightness level.

19. The image processing method according to claim 15, wherein the step of detecting includes the step of:

performing image conversion on a detection-result image that is based on the detected brightness-change pixel using a smoothing filter to exclude a brightness-change pixel that is erroneously detected on the basis of an image obtained as a result of the image conversion.

20. The image processing method according to claim 15, wherein the step of detecting includes the step of:

performing image conversion on a detection-result image that is based on the detected brightness-change pixel using a filter to enlarge a brightness-change pixel region.

21. The image processing method according to claim 20, wherein the step of determining includes the step of:

determining a pixel value of the corresponding pixel in an output image using coefficients by the following equation;

[Rv]=a×

[Dv]+b×

[Mv]

where [Rv] denotes the pixel value of the corresponding pixel in the output image, [Dv] denotes the pixel value of the corresponding pixel in the combined image, [Mv] denotes the pixel value of the corresponding pixel in the intermediate image, and a and b denote the coefficients, wherein a+b=1,

wherein the pixel value of the corresponding pixel in the output image is determined using, as the coefficients, a first set of coefficients used to determine pixel values of a brightness-change pixel region that is detected on the basis of the combined image, and a second set of coefficients to determine pixel values of the brightness-change pixel region enlarged using the filter, the first set of coefficients and the second set of coefficients having different values from each other.

22. The image processing method according to claim 15, wherein the step of generating an intermediate image includes the step of:

applying a smoothing filter to the combined image to generate a blurred image.

23. The image processing method according to claim 15, wherein the step of generating an intermediate image includes the step of:

reducing and enlarging the combined image to generate a blurred image.

24. The image processing method according to claim 15, wherein the step of correcting further includes the step of:

reducing a color saturation of the generated combined image to generate a saturation-reduced image, and

wherein the step of generating an intermediate image generates, upon receiving the generated saturation-reduced image, a blurred image.

25. The image processing method according to claim 15, wherein the step of correcting further includes the step of:

performing an image correction process of applying a low pass filter to an image generated in the step of determining.

26. The image processing method according to any of claims 15 to 25, wherein an exposure period in which the long-time exposure image is generated includes an exposure period in which the short-time exposure image is generated, and

wherein the short-time exposure image and the long-time exposure image are images generated from an identical pixel by a single imaging element.

27. A non-transitory computer storage medium comprising a computer program for causing an image processing apparatus to execute image processing, the computer program comprising the steps of:

causing an image input unit to receive a long-time exposure image and a short-time exposure image;

causing an image analysis unit to detect a brightness-change pixel in which it is determined that a brightness change has occurred during a photographic period corresponding to a long-time exposure time on the basis of analysis of pixel values in the long-time exposure image and the short-time exposure image; and

causing a pixel value correction unit to correct a pixel value of the brightness-change pixel detected in the step of causing an image analysis unit to detect a brightness-change pixel,

wherein the step of causing a pixel value correction unit to correct a pixel value includes the steps of

generating a combined image by selectively combining pixel values in the long-time exposure image and pixel values in the short-time exposure image,

generating a blurred image of the combined image, and

generating an output image by determining a pixel value of the brightness-change pixel detected by the image analysis unit using a pixel value of a corresponding pixel in the combined image and a pixel value of the corresponding pixel in the intermediate image, the corresponding pixel in the combined and intermediate images being located at positions corresponding to the brightness-change pixel.