DEVICE AND ALGORITHM FOR CAPTURING HIGH DYNAMIC RANGE (HDR) VIDEO

• US 20140002694A1
• Filed: 07/02/2012
• Published: 01/02/2014
• Est. Priority Date: 07/02/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: CSR Technology Incorporated (Qualcomm, Inc.)
• Inventors: Levy, Noam, Rapaport, Guy
• Granted Patent: US 9,489,706 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/007 Dynamic range modification applied in cameras using an electronic image sensor H04N23/741, H04N23/743
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G06T 7/35 using statistical methods
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 25/57 Control of the dynamic range

Abstract

Embodiments are directed towards enabling digital cameras to digitally process a captured a Low Dynamic Range image sequence at a real time video rate, and to convert the image sequence into an High Dynamic Range (HDR) image sequence using a pipelined architecture. Two or more image frames are captured using different exposure settings and then combined to form a single HDR output frame in a video sequence. The pipelined architecture operate on adjacent image frames by performing an image alignment, an image mixing, and a tone mapping on the adjacent image frames to generate the HDR image sequence.

Claims

1. A processor based method, comprising:

pipelining a plurality of actions at a real time video rate to generate a high dynamic video output stream by;

capturing a plurality of images as low dynamic range images, at least one image having a different exposure setting than at least one other image in the plurality of images;

performing image alignment between the images in the plurality of images;

creating a single high dynamic range (HDR) image by combining the aligned images; and

encoding the HDR image into a HDR video output stream.

2. The processor based method of claim 1, wherein performing image alignment further comprises:

performing a motion vector extraction between corresponding pixels between the images; and

employing a result of the motion vector extraction transforming the images onto image coordinates of at least one of the images in plurality of images.

3. The processor based method of claim 2, wherein the motion vector extraction further comprises:

employing a motion search using a motion of a block of pixels in at least one of the images to identify an other block of pixels in at least one other image in the plurality based on when a mean reduced sum of absolute differences (MRSAD) is determined to be minimal;

using a result of the motion search to identify a set of correspondences between pixels in at least one of the images with pixels in at least one other image; and

performing a global motion estimation on the set of correspondences to generate an estimated global transformation.

4. The processor based method of claim 3, wherein the motion search further includes performing a corner detection on at least one of the images and at least one other image in the plurality.

5. The processor based method of claim 1, wherein combing the aligned images further comprises:

for each image in the plurality that is represented in a gamma, non-linear, domain, performing a de-gamma transformation on the image;

normalizing the respective image having a brighter exposure setting than an other image in the plurality, to an exposure value of the other darker image;

determining weighting values based in part on luminosity values of the image having the brighter exposure setting; and

using the weighting values to blend dark pixels from the brighter image with bright pixels from the darker image to generate the HDR image.

6. The processor based method of claim 1, wherein pipelining a plurality of actions, further comprises:

performing a tone mapping to reduce a number of bits used to represent a color component in the HDR image, the tone mapping including;

performing a gamma, global mapping on the HDR image; and

performing a non-linear, local mapping that maps a pixel based on a neighbor of pixels'"'"' average luminosity value.

7. The processor based method of claim 1, wherein at least one exposure setting is different in at least one of an exposure time, an f-number, or an ISO level.

8. An image system, comprising:

a image sensing device that is configured to receive a plurality of images as low dynamic range images; and

one or more circuits having a plurality of components thereon operating in a pipeline architecture to perform a plurality of actions at a real time video rate for a sequence of captured images, including;

receiving the plurality of images, at least one image having a different exposure setting when captured than at least one other image in the plurality of images;

performing image alignment between the images;

creating a single high dynamic range image by combining the aligned images using a pixel-wise blending; and

encoding the high dynamic range image into a high dynamic range video output stream.

9. The image system of claim 8, wherein performing image alignment further comprises:

performing a motion vector extraction between corresponding pixels between the images; and

employing a result of the motion vector extraction transforming the images onto image coordinates of at least one image in the plurality.

10. The image system of claim 9, wherein the motion vector extraction further comprises:

employing a motion search using a motion of a block of pixels in at least one of the images to identify an other block of pixels in at least one other image based on when a mean reduced sum of absolute differences (MRSAD) is determined to be minimal;

using a result of the motion search to identify a set of correspondences between pixels in the images; and

performing a global motion estimation on the set of correspondences to generate an estimated global transformation.

11. The image system of claim 10, wherein the motion search further includes performing a corner detection on the images.

12. The image system of claim 8, wherein blending further comprises for saturated areas within one of the images, using pixels from a darker image in the plurality to recover burned out details in a scene when creating the HDR image.

13. The image system of claim 8, wherein the different exposure setting is different in at least one of an exposure time, an f-number, or an ISO level.

14. The image system of claim 8, wherein the pipelined plurality of actions, further comprises:

performing a tone mapping to reduce a number of bits used to represent a color component in the HDR image, the tone mapping including;

performing a gamma, global mapping on the HDR image; and

performing a non-linear, local mapping that maps a pixel based on a neighbor of pixels'"'"' average luminosity value.

15. A storage device having stored thereon a plurality of computer-executable instructions that when installed on a digital camera performs actions, comprising:

performing within a pipelined architecture using the computer-executable instructions to perform actions for images within a sequence of images, including;

capturing a first image with a first exposure setting;

capturing a second image with a second exposure setting that is different from the first exposure setting, the first image and the second image being captured as low dynamic range images;

performing image alignment between the first image and the second image;

creating a high dynamic range (HDR) image by blending the aligned first image and the second image using a pixel-wise blending; and

encoding the HDR image into a HDR video output stream.

16. The storage device of claim 15, wherein performing image alignment further comprises:

performing a motion vector extraction between corresponding pixels between the first image and the second image; and

employing a result of the motion vector extraction transforming one of the first image or the second image onto image coordinates of the other of the second image or the first image.

17. The storage device of claim 16, wherein the motion vector extraction further comprises:

employing a motion search using a motion of a block of pixels in the first image to a identify an other block of pixels in the second image based on when a mean reduced sum of absolute differences (MRSAD) is determined to be minimal;

using a result of the motion search to identify a set of correspondences between pixels in the first image with pixels in second image; and

performing a global motion estimation on the set of correspondences to generate an estimated global transformation.

18. The storage device of claim 15, wherein blending further comprises for saturated areas within one of the first image or second image, using pixels from a darker of the first image or the second image to recover burned out details in a scene when creating the HDR image.

19. The storage device of claim 15, wherein the first exposure setting and the second exposure setting are different in at least one of an exposure time, an f-number, or an ISO level.

20. The storage device of claim 15, wherein the pipelined plurality of actions, further comprises:

performing a tone mapping to reduce a number of bits used to represent a color component in the HDR image, the tone mapping including;

performing a gamma, global mapping on the HDR image; and

performing a non-linear, local mapping that maps a pixel based on a neighbor of pixels'"'"' average luminosity value.