System and process for generating high dynamic range video

• US 20050047676A1
• Filed: 10/15/2004
• Published: 03/03/2005
• Est. Priority Date: 04/29/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kang, Sing Bing, Uyttendaele, Matthew T., Szeliski, Richard, Winder, Simon
• Granted Patent: US 7,010,174 B2
• Time in Patent Office: Days
• US Class Current: 382/274
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

A system and process for generating High Dynamic Range (HDR) video is presented which involves first capturing a video image sequence while varying the exposure so as to alternate between frames having a shorter and longer exposure. The exposure for each frame is set prior to it being captured as a function of the pixel brightness distribution in preceding frames. Next, for each frame of the video, the corresponding pixels between the frame under consideration and both preceding and subsequent frames are identified. For each corresponding pixel set, at least one pixel is identified as representing a trustworthy pixel. The pixel color information associated with the trustworthy pixels is then employed to compute a radiance value for each pixel set to form a radiance map. A tone mapping procedure can then be performed to convert the radiance map into an 8-bit representation of the HDR frame.

Claims

1. A computer-implemented process for generating high dynamic range (HDR) video from an image sequence captured while varying the exposure of each frame, said process comprising using a computer to perform the following process actions:

setting the exposure for each frame of the video image sequence prior to it being captured as a function of the pixel brightness distribution;

for each frame of the video after a prescribed number of initiating frames,

identifying corresponding pixels between the frame under consideration and both preceding and subsequent neighboring frames representing a corresponding pixel set,

for each set of corresponding pixels, identifying at least one pixel in the set that represents a trustworthy pixel and employing the pixel color information associated with the one or more identified trustworthy pixels to compute a radiance value for that set of pixels to form a radiance map representing a frame of the HDR video.

2. The process of claim 1, further comprising a process action of tone mapping the radiance map to convert it into an 8-bit representation suitable for rendering.

3. The process of claim 1, wherein the frames of the video image sequence alternate between one captured at a shorter exposure and one captured at a longer exposure.

4. The process of claim 3, wherein the process action of setting the exposure for each frame of the video image sequence comprises an action of setting the exposures of each frame so that the exposure ratio between each sequential pair of long and short exposure frames is minimized while simultaneously producing a substantially full range of radiances for the scene depicted in the frames.

5-30. -30. (cancelled)

31. A system for generating high dynamic range (HDR) video from a video image sequence captured while varying the exposure of each frame, comprising:

a video camera used to capture said video image sequence;

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

computing an exposure for each frame of the video image sequence prior to it being captured as a function of the pixel brightness distribution in frames previously captured by the video camera and causing the video camera to capture each frame at the exposure computed for that frame, wherein said computed exposure is made to alternate between a shorter exposure and a longer exposure;

for each frame of the video obtained from the video camera after a prescribed number of initiating frames,

identifying corresponding pixels between the frame under consideration and both preceding and subsequent neighboring frames representing a corresponding pixel set,

for each set of corresponding pixels, identifying at least one pixel in the set that represents a trustworthy pixel and employing the pixel color information associated with the one or more identified trustworthy pixels to compute a radiance value for that set of pixels to form a radiance map representing a frame of the HDR video.

32-54. -54. (cancelled)

55. A computer-readable medium having computer-executable instructions for generating high dynamic range (HDR) video from a video image sequence captured while varying the exposure of each frame, said computer-executable instructions comprising:

setting the exposure for each frame of the video image sequence prior to it being captured as a function of the pixel brightness distribution;

for each frame of the video after a prescribed number of initiating frames,

identifying corresponding pixels between the frame under consideration and both preceding and subsequent neighboring frames representing a corresponding pixel set, and

for each set of corresponding pixels, identifying at least one pixel in the set that represents a trustworthy pixel and employing the pixel color information associated with the one or more identified trustworthy pixels to compute a radiance value for that set of pixels to form a radiance map representing a frame of the HDR video, and

tonemapping the radiance map to convert it into an 8-bit representation suitable for rendering.

56-60. -60. (cancelled)