Methods and Systems for Enhanced Dynamic Range Images and Video from Multiple Exposures

• US 20100183071A1
• Filed: 01/19/2009
• Published: 07/22/2010
• Est. Priority Date: 01/19/2009
• Status: Active Grant

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Rubenstein, Ron, Zhao, Jie, Segall, Christopher A.
• Granted Patent: US 8,406,569 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Aspects of the present invention relate to systems and methods for video coding and decoding. Some aspects relate to coding and decoding of enhanced dynamic range video sequences based on a series of low-dynamic-range (LDR) frames, captured at a first exposure configuration, that are multi-plexed with a series of enhancement frames that are captured at a second exposure configuration.

Claims

1. A method for decoding a video sequence, said method comprising:

a) receiving an interleaved frame sequence comprising a plurality of low-dynamic-range (LDR) reference frames captured with a first exposure configuration and a plurality of enhancement frames captured with a second exposure configuration;

b) separating a luminance channel from at least one chrominance channel in said LDR reference frames and said at least one enhancement frame;

c) performing motion estimation on said luminance channels of said LDR reference frames and said enhancement frames;

d) performing motion compensation on said luminance channels of said LDR reference frames and said enhancement frames to align said enhancement frames with said LDR reference frames;

e) interpolating between said LDR reference frames to create interpolated LDR reference frames for times when said enhancement frames were captured;

f) interpolating between said enhancement frames to create warped enhancement frames for times when said LDR reference frames were captured;

g) assigning temporal weight values to said enhancement frames;

h) creating a mis-match map identifying where said enhancement frames are misaligned with said LDR reference frames;

i) fusing the luminance channels of said enhancement frames, said warped enhancement frames, said interpolated LDR reference frames and said LDR reference frames;

j) fusing the chrominance channels of said enhancement frames, said warped enhancement frames, said interpolated LDR reference frames and said LDR reference frames;

k) performing tone mapping on said fused luminance channels to conform channel values with a valid intensity range; and

l) combining said fused, tone-mapped luminance channels with said fused chrominance channels to produce a reconstructed sequence.

2. A method as described in claim 1 wherein said fusing the luminance channels of said enhancement frames comprises computing an adaptive Laplacian pyramid for said enhancement frames.

3. A method as described in claim 1 wherein said fusing the luminance channels of said enhancement frames comprises merging said adaptive Laplacian pyramids using pixel-wise maximization.

4. A method for generating an enhanced dynamic range (EDR) image frame from a low dynamic range (LDR) image frame and a plurality of enhancement frames, said method comprising:

a) computing a Laplacian pyramid for each of said enhancement frames and said LDR image frame;

b) computing blend pyramids corresponding to said enhancement frame pyramids, said blend pyramids denoting how pixels in said enhancement frame pyramids are to be used in a pyramid blending process; and

c) performing pixel-wise maximization between said Laplacian pyramids according to said blend pyramids, thereby creating a blended pyramid.

5. A method as described in claim 4 wherein said blend pyramids comprise weighting factors that define a relative contribution of pixels in said enhancement frame pyramids during said performing.

6. A method as described in claim 4 further comprising assigning temporal weight values to said enhancement frame pyramids.

7. A method as described in claim 4 further comprising performing a binary thresholding operation on said blend pyramids.

8. A method as described in claim 4 further comprising performing a smoothing operation on said blend pyramids.

13. A method as described in claim 4 wherein said legacy device adds skip frames between said LDR reference frames at time instants when said enhancement frames occur.

9. A method for fusing chrominance channels from a low dynamic range (LDR) image frame and a plurality of enhancement frames, said method comprising:

a) computing a color fusion map for each of said enhancement frames, wherein said computing a color fusion map comprises

i) computing a smooth border map; and

ii) normalizing said smooth border map such that the larger of said smooth border maps has a value of one;

iii) adjusting values of said normalized smooth border map based on a degree of mis-match between said enhancement frames and said LDR image frame;

b) computing a saturation confidence map, wherein values in said saturation confidence map are related to the difference between its corresponding luminance value and a mid-tone value; and

c) fusing said chrominance channels based on said color fusion map and said saturation confidence map.

10. A method for tone-mapping a higher-dynamic-range (HDR) image to lower-dynamic-range (LDR) image, said method comprising:

a) computing a Laplacian pyramid for said HDR image;

b) applying a compression map to a coarsest level of said Laplacian pyramid, thereby creating a compressed pyramid; and

c) reconstructing said compressed pyramid.

11. A method for detecting misregistration between differently-exposed images, said method comprising:

a) receiving a first image that has been captured with a first exposure configuration;

b) receiving a second image that has been captured with a second exposure configuration;

c) performing exposure compensation on said first image relative to said second image;

d) computing a Laplacian pyramid for each of said first image and said second image;

e) calculating a local normalized correlation between pixels in said first image and co-located pixels in said second image;

f) calculating a local activity for said first image and said second image; and

g) merging levels of said Laplacian pyramid using a mis-match map based on said local normalized correlation and said local activity.

12. A method for capturing and encoding an enhanced dynamic range video sequence, said method comprising:

a) capturing a plurality of low-dynamic-range (LDR) reference frames at a substantially fixed interval using a first exposure configuration;

b) capturing at least one enhanced frame using a second exposure configuration; and

c) encoding said LDR reference frames and said at least one enhanced frame such that a legacy device can decode said LDR reference frames into a video sequence without reference to said enhanced frame and such that a supporting decoder can decode said LDR reference frames and said at least one enhancement frame into an enhanced video sequence.

14. A method for capturing and encoding an enhanced dynamic range video sequence, said method comprising:

a) capturing a plurality of first-configuration reference frames using a first exposure configuration;

b) generating first exposure configuration metadata that identifies first exposure configuration parameters;

c) capturing at least one second-configuration frame using a second exposure configuration;

d) generating second exposure configuration metadata that identifies second exposure configuration parameters;

e) performing exposure compensation on said first-configuration reference frames and said at least one second-configuration frame to correct a difference between said first exposure configuration parameters and said second exposure configuration parameters, thereby creating a compensated video sequence;

f) encoding said compensated video sequence; and

g) associating said compensated video sequence with said first exposure configuration metadata and said second exposure configuration metadata.

15. A method for decoding an enhanced dynamic range video sequence, said method comprising:

a) receiving a compensated video sequence comprising a plurality of compensated first-configuration reference frames and at least one compensated second-configuration frame, which are associated with first exposure configuration metadata related to said first-configuration reference frames and second exposure configuration metadata related to said at least one second-configuration frame;

b) decoding said compensated first-configuration reference frames and said at least one compensated second-configuration frame, thereby creating a decoded compensated sequence; and

c) performing inverse exposure compensation on said decoded compensated sequence.

16. A method for enabling weighted prediction in a video sequence, said method comprising:

a) receiving a first video frame that has been captured with a first exposure configuration;

b) receiving first video frame exposure metadata that describes first exposure configuration parameters;

c) determining a first brightness value for said first video frame based on said first video frame exposure metadata;

d) receiving a second video frame that has been captured with a second exposure configuration;

e) receiving second video frame exposure metadata that describes second exposure configuration parameters;

f) determining a second brightness value for said second video frame based on said first video frame exposure metadata;

g) enabling weighted prediction for said first video frame and said second video frame based on a relationship between said first brightness value and said second brightness value.