MULTIPLE EXPOSURE HIGH DYNAMIC RANGE IMAGE CAPTURE

• US 20100271512A1
• Filed: 04/20/2010
• Published: 10/28/2010
• Est. Priority Date: 04/23/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: CSR Technology Incorporated (Qualcomm, Inc.)
• Inventors: Garten, Haim
• Granted Patent: US 8,237,813 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G09G 2320/02 Improving the quality of display appearance
• CPC Class Codes: G09G 2320/0285 using tables for spatial correction of display data
• CPC Class Codes: G09G 2340/0414 Vertical resolution change
• CPC Class Codes: G09G 2340/0421 Horizontal resolution change
• CPC Class Codes: G09G 2340/06 Colour space transformation
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 5/06 using colour palettes, e.g. look-up tables
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Techniques for creating a High Dynamic Range (HDR) image within a consumer grade digital camera from a series of images of a scene captured at different exposure levels, and displaying the HDR image on the camera'"'"'s built-in display, are provided. The approach employs mixing images of the series to incorporate both scene shadow and highlight details, and the removing of “ghost” image artifacts appearing in the mixed HDR image resulting from movement in the scene over the time the series images are captured. The low computational resource utilization of the present invention'"'"'s image mixing and ghost removal processing operations, along with the present invention'"'"'s ability to commence image mixing and ghost removal prior to the acquisition of all series images, can significantly reduce the time required to generate and display a tone mapped HDR image.

Claims

1. A method of mixing a series of two or more images of a scene, each series image composed of pixels containing digital bits to generate a composite image in which each pixel contains a number of digital bits, the number being greater than the number of digital bits contained in any series image pixel, comprising:

capturing the series images at different exposure levels;

registering the counterpart pixels of each image of the image series one to another;

deriving a normalized image exposure level for each image in the series by using the exposure level of the series image that is least exposed as a reference level;

employing the normalized image exposure levels in an image blending process;

using the image blending process to blend together a first image of the series and a second image of the series, wherein the second image of the series is captured at a higher exposure level than the first image of the series, to generate an intermediate image, the intermediate image being a mixed output image when the series is composed of two images; and

repeating-the image blending process using the generated intermediate image in place of the first image of the series and a subsequent series image in place of the second image of the series when the series is composed of more than two images, each successively blended series image being exposed at a next higher exposure level than the previously blended series image, until all series images have been blended, the last generated intermediate image being the mixed output image.

2. The method of claim 1 wherein the series is composed of three images and the series image captured at an exposure level greater than one of the other two series images, and less than the other one of the two series images, is captured prior to the capture of either of the other two series images.

3. The method of claim 1 wherein the image blending process blends the counterpart pixels of two images, the second image being exposed at a higher exposure level then the first, and includes:

deriving a luma value for each pixel in the second image;

using the luma value of a second image pixel as an index into a look-up table to obtain a weighting value between the numbers zero and unity;

using the weighting value, the normalized exposure level of the second image, and the second image pixel to generate a processed second image pixel;

selecting a first image pixel that corresponds to the second image pixel;

using the first image pixel and the result of subtracting the weighting value from unity, to generate a processed first image pixel;

adding the processed first image pixel to the counterpart processed second image pixel to generate a blended image pixel;

repeating the above processing sequence until each second image pixel has been blended with its counterpart first image pixel.

4. The method of claim 3 wherein the obtained weighting value decreases as the luma value used as the index into the look-up table increases.

5. The method of claim 3 wherein a different look-up table is used for each image for obtaining the weighting value.

6. The method of claim 1 wherein image mixing commences prior to the capture of all the images that comprise the image series.

7. The method of claim 1 wherein image mixing commences immediately after the capture of the second image of the series.

8. A method of removing location shifted replications of scene objects appearing in mixed image data generated by a digital image mixing process applied to a series of scene images acquired at different exposure levels and different times, such scene images having their counterpart pixels registered one to another, comprising:

selecting a reference image from one of the acquired scene images;

determining a variance of the luma of each pixel of the mixed image data, as compared to its counterpart reference image data pixel;

replacing each pixel of the mixed image data with a variance exceeding a first threshold level with its counterpart reference image data pixel, the resulting first processed mixed image data being the output of a first stage of processing;

calculating the difference between the luma of each first processed mixed image data pixel and the luma of its counterpart reference image data pixel;

generating a second threshold level based on the maximum value of the differences; and

replacing each first processed mixed image data pixel exceeding the second threshold level with its counterpart reference image data pixel, the resulting second processed mixed image data being the output of a second stage of processing.

9. The method of claim 8 wherein the reference image is the acquired scene image with the lowest exposure level.

10. The method of claim 8 wherein the digital image mixing process is applied to a series of three scene images, the exposure level of a first series image being lower than the exposure level of a second series image and the exposure level of the second series image being lower than the exposure level of a third series image, the reference image being the second series image.

11. The method of claim 8 wherein the digital image mixing process is applied to three series images, the exposure level of a first series image being lower than the exposure level of a second series image and the exposure level of the second series image being lower than the exposure level of a third series image, comprising:

determining the regions of the second series image that correspond to mixed image data with variances that exceed the first threshold;

calculating the total area of the regions;

calculating a sum of areas of the regions that are saturated and a ratio of the sum of saturated area to total area;

selecting as reference the second series image when the ratio is less than or equal to 0.03 to 1; and

selecting as reference the first series image when the ratio is greater than 0.03 to 1.

12. The method of claim 8 wherein a first reference image is used for the first stage of processing and a second reference image is used for the second stage of processing.

13. An image capturing device that captures a series of two or more digital images of a scene at different exposure levels, each series image composed of pixels containing a number of digital bits, that includes:

an image registration processor that registers the counterpart pixels of each image of the image series one to another;

an image mixer that combines multiple images of the series, such series images having their counterpart pixels registered one to another, to generate a single image whose pixels contain a number of digital bits, the number being greater than the number of digital bits contained in any series image pixel, wherein the mixer is comprised of;

an image normalizer that normalizes the image exposure level for each image in the series by using the exposure level of the least exposed series image as a reference level;

an image blender, that uses the normalized exposure level to blend together a first image of the series and a second image of the series, wherein the second image of the series is captured at a higher exposure level than the first image of the series, and generates a first intermediate image, the first intermediate image being the mixed output image from the blender when the series is composed of two images;

an image blender to blend together the first intermediate image with a third image of the series, the third image of the series being exposed at a next higher exposure level, to generate a subsequent intermediate image, when the series is composed of more than two images; and

an image blender to generate subsequent intermediate images by blending each derived intermediate image with a series image, each successively blended series image being exposed at a higher exposure level than the previously blended series image, until all series images have been blended, the last generated intermediate image being the mixed output image from the blender.

14. The image capturing device of claim 13 that mixes a series of captured images composed of three captured images and the series image exposed at an exposure level greater than one of the other two series images, and less than the other one of the two series images is captured prior to the capture of either of the other two series images.

15. The image capturing device of claim 13 that blends the pixels of two images, the second image being exposed at a higher exposure level then the first image, that includes:

a luma conversion circuit into which each pixel in the image exposed at the second image exposure level is input, and that outputs the luma value of each input pixel;

a look-up table that outputs a weighting value between the numbers zero and unity for each input luma value;

a processing circuit that generates as output a second image processed image pixel from the input second image pixel, the normalized exposure level of the second image, and the input weighting value;

a processing circuit that generates as output a first image processed pixel from the input first image pixel and the input resulting from subtracting the weighting value from unity;

an adder circuit that adds the input first image processed pixel to the input counterpart second image processed pixel and generates a blended image pixel;

16. The image capturing device claim 15 that outputs a lower weighting value as the input luma value to the look-up table increases.

17. The image capturing device of claim 15 that uses a first look look-up table for the first image and a second look-up table for the second image.

18. The image capturing device of claim 13 that begins mixing the multiple series images prior to the capture of all the images that comprise the image series.

19. The image capturing device of claim 13 that begins mixing the multiple images of the series immediately after the capture of the second image of the series.

20. An image capturing device that captures a series of two or more digital images of a scene at different exposure levels and different times, each series image composed of pixel data containing a number of digital bits, that includes:

an image registration processor that registers the counterpart pixels of each image of the image series one to another;

an image mixer that combines multiple images of the series to generate as output mixed image data whose pixels contain a number of digital bits, the number being greater than the number of digital bits contained in any series image pixel and;

a ghost remover that removes location shifted replications of scene objects appearing in the mixed image data output, wherein the ghost remover is comprised of;

a reference image selection circuit that selects a reference image from one of the acquired scene images;

a variance calculation circuit that calculates a variance of the luma of each pixel of the mixed image data, as compared to its counterpart reference image data pixel;

a first pixel replacement circuit that replaces each mixed image data pixel with a variance that exceed a first threshold level, with its counterpart reference image data pixel, a first processed mixed image data being the output of the first pixel replacement circuit, the output being the result of a first stage of processing;

a comparison circuit that calculates the difference between the luma of each first processed mixed image data pixel and the luma of its counterpart reference image data pixel, and generates a second threshold level based on the maximum value of the differences; and

a second pixel replacement circuit that replaces each first processed mixed image data pixel exceeding the second threshold level with its counterpart reference image data pixel, a resulting second processed mixed image data being the output of a second stage of processing.

21. The image capturing device of claim 20 that selects the captured series image with the lowest exposure level as the reference image.

22. The image capturing device of claim 20 wherein a series of three digital images are captured, a first series image with an exposure level lower than the exposure level of a second series image and the second series image with an exposure level lower than the exposure level of a third series image, the reference image selected being the second series image.

23. The image capturing device of claim 20 wherein the digital image mixer mixes three series images, the exposure level of a first series image being lower than the exposure level of a second series image and the exposure level of the second series image being lower than the exposure level of a third series image, and the ghost remover reference image selection circuit includes:

a region determination circuit that determines the regions of the second series image that correspond to mixed image data with variances that exceed the first threshold; and

a calculation circuit that outputs the total area of the regions, the sum of areas of the regions that are saturated, and the ratio of the sum of saturated area to total area;

the reference image selection circuit selecting as reference the second scene image when the ratio is less than or equal to 0.03 to 1; and

the reference image selection circuit selecting as reference the first scene image when the ratio is greater than 0.03 to 1.

24. The image capturing device of claim 20 that selects a first reference image for the first stage of processing and a second reference image for the second stage of processing.

25. A digital camera that captures a series of two or more digital images of a scene at different exposure levels, and at different times, and generates therefrom a tone mapped high dynamic range image, comprising:

an image registration processor that registers the counterpart pixels of each image of the image series one to another;

an image mixer that combines multiple images of the series to generate a single image whose pixels contain a number of digital bits, the number being greater than the number of digital bits contained in any series image pixel, wherein the mixer is comprised of;

an image normalizer that normalizes the image exposure level for each image in the series by using the exposure level of the least exposed image of the series as the reference level;

an image blender, that uses the normalized exposure level to blend together a first image of the series and a second image of the series, wherein the first image of the series is captured at a lower exposure level than the second image of the series, and generates a first intermediate image, the first intermediate image being the mixed output image from the blender when the series is composed of two images;

an image blender to blend together the first intermediate image with a third image of the series, the third image of the series being exposed at a next higher exposure level, to generate a subsequent intermediate image, when the series is composed of more than two images; and

an image blender to generate subsequent intermediate images by blending each derived intermediate image with a series image, each successively blended series image being exposed at a higher exposure level than the previously blended series image, until all series images have been blended, the last generated intermediate image containing output mixed image data;

a ghost remover that removes location shifted replications of scene objects appearing in the output mixed image data, wherein the ghost remover is comprised of;

a reference image selection circuit that selects a reference image from one of the acquired series images;

a variance calculation circuit that calculates a variance of the luma of every pixel of the mixed image data, as compared to pixels of the reference image;

a first pixel replacement circuit that replaces pixels of the mixed image data with variances that exceed a first threshold value, with counterpart reference image pixels, first processed mixed image data pixels being the output of the first pixel replacement circuit, the output being the result of a first stage of processing;

a comparison circuit that compares the luma of the first processed mixed image data pixels with the luma of counterpart reference image pixels, the comparison circuit calculating the differences between the luma of the first processed mixed image data pixels and the luma of the pixels of the reference image, and generates a second threshold value based on the maximum value of the differences; and

a second pixel replacement circuit that replaces first processed mixed image data pixels exceeding the second threshold value with counterpart reference image pixels, the resulting second processed mixed image data being the output of a second stage of processing;

and

a tone mapping processor that maps the second process mixed image data pixels to image data pixels that contain the number of digital bits that can be presented on the digital camera'"'"'s built-in digital image display.

26. The digital camera of claim 25 wherein the image registration processor, the image mixer, the ghost remover, or the tone mapping processor are implemented by the use of one or more programmable general purpose processors.

27. The digital camera of claim 25 wherein the image registration processor, the image mixer, the ghost remover, or the tone mapping processor are implemented by the use of one or more programmable application specific processors.

28. The digital camera of claim 25 wherein the image registration processor, the image mixer, the ghost remover, or the tone mapping processor are implemented by the use of one or more fixed function application specific processors.

29. A method of mixing a series of two or more images of a scene, each series image composed of pixels containing digital bits to generate a composite image in which each pixel contains a number of digital bits, the number being greater than the number of digital bits contained in any series image pixel, comprising:

capturing the series images at different exposure levels;

registering the counterpart pixels of each image of the image series one to another;

deriving a normalized image exposure level for each image in the series by using the exposure level of the series image that is least exposed as a reference level;

employing the normalized image exposure levels in an image blending process;

using the image blending process to blend together a first image of the series and a second image of the series, wherein the second image of the series is captured at a higher exposure level than the first image of the series, to generate a first intermediate image, the first intermediate image being a mixed output image when the series is composed of two images;

using the image blending process to blend together the first intermediate image with a third image of the series, the third image of the series being exposed at a next higher exposure level, to generate a second intermediate image, when the series is composed of more than two images; and

using the image blending process to generate subsequent intermediate images by blending each generated intermediate image with a series image, each successively blended series image being exposed at a next higher exposure level than the previously blended series image, until all series images have been blended, the last generated intermediate image being the mixed output image.