Combined HDR/LDR video streaming

• US 10,805,505 B2
• Filed: 04/05/2019
• Issued: 10/13/2020
• Est. Priority Date: 02/12/2016
• Status: Active Grant

• Current Assignee: Contrast, Inc.
• Original Assignee: Contrast, Inc.
• Inventors: Kiser, Willie C., Tocci, Nora, Tocci, Michael D.
• Primary Examiner(s): Kumar, Pankaj
• Assistant Examiner(s): Newlin, Timothy R
• Application Number:
  US16/376,149
• Publication Number:
  US 20190238726A1
• Time in Patent Office: 557 Days
• CPC Class Codes: G02B 27/1013 for colour or multispectral image sensors, e.g. splitting an image into monochromatic image components on respective sensors spectral imaging systems G01J
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 1/00068 Calculating or estimating
• CPC Class Codes: H04N 1/6027 Correction or control of colour gradation or colour contrast H04N1/6058 takes precedence
• CPC Class Codes: H04N 1/603 controlled by characteristics of the picture signal generator or the picture reproducer
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
• CPC Class Codes: H04N 9/67 for matrixing camera processing pipelines for matrixing of colour signals H04N23/85

Abstract

The invention provides methods for broadcasting video in a dual HDR/LDR format such that the video can be displayed in real time by both LDR and HDR display devices. Methods and devices of the invention process streams of pixels from multiple sensors in a frame-independent manner to produce an HDR video signal in real time. That HDR video signal is then tone-mapped to produce an LDR video signal, the LDR signal is subtracted from the HDR signal to calculate a residual signal, and the LDR signal and the residual signal are merged into a combined signal that is broadcast via a communications network.

Claims

1. A method for streaming video for high dynamic range (HDR) and low dynamic range (LDR) display, the method comprising:

receiving frame-independent pixel values from multiple image sensors on an HDR video camera and producing a ≥

12-bit HDR signal comprising at least 12 bits for dynamic range;

tone-mapping the HDR signal to produce an 8-bit LDR signal;

subtracting the 8-bit LDR signal from the ≥

12-bit HDR signal to yield a residual signal, wherein the residual signal consists of the result of subtracting the 8-bit LDR signal from the ≥

12-bit HDR signal;

produce an output signal comprising the 8-bit LDR signal and the residual signal; and

transmitting the output signal for HDR and LDR display.

2. The method of claim 1, wherein the output signal is displayable as an 8-bit video by an LDR display, and displayable as HDR video by an HDR display.

3. The method of claim 1, wherein the HDR signal is produced from the pixel values by a pipeline that includes

a kernel operation that identifies saturated pixel values;

and

a merge operation that merges the pixel values to produce the HDR signal.

4. The method of claim 3, the multiple image sensors all capture images simultaneously through a single lens.

5. The method of claim 4, further comprising receiving incoming light through the lens and splitting the light via at least one beamsplitter onto the multiple image sensors, wherein at least 95% of the light gathered by the imaging lens is captured by the multiple image sensors.

6. The method of claim 4, wherein the multiple image sensors include at least a high exposure (HE) sensor and a middle exposure (ME) sensor, and wherein merging the sequences includes using HE pixel values that are not saturated and ME pixel values corresponding to the saturated pixel values.

7. The method of claim 4, further comprising beginning to merge portions of the pixel values while still streaming later-arriving pixel values through the kernel operation.

8. The method of claim 7, wherein the pipeline is provided by a processing device selected from the group consisting of a field-programmable gate array and an application-specific integrated circuit, and further wherein each of the image sensors comprises a color filter array, and the method further comprises demosaicing the HDR signal after the merging.

9. The method of claim 8, wherein multiple image sensors capture images that are optically identical except for light level.

10. The method of claim 3, further comprising adjusting, in the pipeline, one or more pixel values from one of the multiple sensors towards a color correction space of another one of the multiple sensors, wherein the adjusting involves changing the one or more pixel values away from true color and is performed prior to the merge operation.

11. The method of claim 10, further comprising color correcting the HDR signal prior to the tone-mapping step.

12. The method of claim 1, wherein the output signal consists of the 8-bit LDR signal and the residual signal.