Green image data processing

• US 10,582,168 B2
• Filed: 06/26/2018
• Issued: 03/03/2020
• Est. Priority Date: 02/14/2013
• Status: Active Grant

• Current Assignee: Red.com, LLC
• Original Assignee: Red.com, LLC
• Inventors: Jannard, James H., Nattress, Thomas Graeme, Greene, Richard, Mathur, Uday
• Primary Examiner(s): Chen, Chiawei
• Application Number:
  US16/018,849
• Publication Number:
  US 20190124302A1
• Time in Patent Office: 616 Days
• CPC Class Codes: H04N 19/426 using memory downsizing methods
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/531 being rotatable or detachable
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

Embodiments provide a video camera that can be configured to highly compress video data in a visually lossless manner. The camera can be configured to transform blue, red, and/or green image data in a manner that enhances the compressibility of the data. The camera can be configured to transform at least a portion of the green image data in a manner that enhances the compressibility of the data. The data can then be compressed and stored in this form. This allows a user to reconstruct the red, blue, and/or green image data to obtain the original raw data or a modified version of the original raw data that is visually lossless when demosaiced. Additionally, the data can be processed in a manner in which at least some of the green image elements are demosaiced first and then the red, blue, and/or some green elements are reconstructed based on values of the demosaiced green image elements.

Claims

1. A portable electronic device capable of capturing digital motion video, the portable electronic device comprising:

at least one memory device;

one or more image sensors configured to convert light incident on the one or more image sensors into mosaiced color motion video image data, the mosaiced color motion video image data comprising a plurality of color channels, the color channels comprising at least a first green color channel, a second green color channel, a red color channel, and a blue color channel; and

one or more hardware processors configured to, for a plurality of motion video image frames;

for each respective green picture element of a plurality of green picture elements of the second green color channel, modify an initial value corresponding to the respective green picture element using a calculated value derived from values of a first kernel, the first kernel comprising a plurality of green picture elements of the first green color channel that are selected based on a position of the respective green picture element, to generate transformed second green color channel data;

for each respective red picture element of a plurality of red picture elements of the red color channel, modify an initial value corresponding to the respective red picture element using a calculated value derived from values of a second kernel, the second kernel comprising a plurality of green picture elements of one or more of the first green color channel and the second green color channel that are selected based on a position of the respective red picture element, to generate transformed red color channel data;

for each respective blue picture element of a plurality of blue picture elements of the blue color channel, modify an initial value corresponding to the respective blue picture element using a calculated value derived from values of a third kernel, the third kernel comprising a plurality of green picture elements of one or more of the first green color channel and the second green color channel that are selected based on a position of the respective blue picture element, to generate transformed blue color channel data;

compress the transformed second green color channel data, data corresponding to the first green color channel, the transformed red color channel data, and the transformed blue color channel data, to generate compressed mosaiced color motion video image data; and

store the compressed mosaiced color motion video image data in the memory device.

2. The portable electronic device of claim 1, wherein the one or more hardware processors are configured to, for each respective green picture element of the plurality of green picture elements of the second green color channel, perform the modification of the initial value corresponding to the respective green picture element by subtracting the calculated value from the initial value.

3. The portable electronic device of claim 2, wherein, for each respective green picture element of the plurality of green picture elements of the second green color channel, the calculated value comprises an average of the values of the plurality of green picture elements in the first kernel.

4. The portable electronic device of claim 3, wherein the plurality of green picture elements in the first kernel comprise at least two green picture elements which are on opposite sides of the respective green picture element.

5. The portable electronic device of claim 3, wherein the plurality of green picture elements in the first kernel include green picture elements from two picture element rows or from four picture element rows.

6. The portable electronic device of claim 1 further comprising:

a lens mount; and

a removable lens attachable to the lens mount,

wherein the mosaiced color motion video image data is mosaiced according to a Bayer pattern.

7. The portable electronic device of claim 1, wherein the one or more hardware processors are further configured to:

for each respective green picture element of the plurality of green picture elements of the second green color channel, determine whether the first kernel comprising the plurality of green picture elements of the first green color channel comprises any virtual green picture elements positioned beyond an edge of the one or more image sensors, and if so, to assign a value to each of the virtual green picture elements for use in deriving the calculated value.

8. The portable electronic device of claim 7, wherein the value assigned to each of the virtual green picture elements is a copy of the value of a non-virtual green picture element in the first kernel.

9. The portable electronic device of claim 1, wherein the plurality of green picture elements of the first kernel comprising the plurality of green picture elements of the first green color channel include at least four green picture elements of the first green color channel that form a square centered on the respective green picture element of the plurality of green picture elements of the second green color channel.

10. The portable electronic device of claim 9, wherein the first kernel comprising the plurality of green picture elements of the first green color channel has four green picture elements.

11. The portable electronic device of claim 9, wherein the first kernel comprising the plurality of green picture elements of the first green color channel comprises at least 16 green picture elements.

12. The portable electronic device of claim 1, wherein for each respective green picture element of the plurality of green picture elements of the second green color channel, the first kernel comprising the plurality of green picture elements of the first green color channel is centered on the respective green picture element.

13. The portable electronic device of claim 1, wherein:

the generation of the transformed second green color channel data results in a spatial decorrelation of the data corresponding to the first green color channel from the transformed second green color channel data,

the generation of the transformed red color channel data results in a spatial decorrelation of data corresponding to the first green color channel from the transformed red color channel data, and

the generation of the transformed blue color channel data results in a spatial decorrelation of the data corresponding to the first green color channel from the blue color channel data.

14. The portable electronic device of claim 1, wherein the compression comprises application of a lossy compression algorithm, and wherein the one or more hardware processors are further configured to, for the plurality of image frames, apply a non-linear pre-emphasis function to the mosaiced color motion video image data prior to the compression.

15. The portable electronic device of claim 14, wherein the one or more hardware processors are configured to apply the pre-emphasis function to at least the transformed red color channel data, the transformed blue color channel data, and the transformed second green color channel data.

16. A portable electronic device capable of capturing digital motion video, the portable electronic device comprising:

one or more image sensors; and

one or more hardware processors configured to, for a plurality of motion video image frames;

receive mosaiced color motion video image data acquired by the one or more image sensors, the mosaiced color motion video image data comprising a plurality of picture element values for each of a plurality of color channels, the color channels comprising at least a first green color channel, a second green color channel, a red color channel, and a blue color channel;

for each respective green picture element of a plurality of green picture elements of the second green color channel, modify an initial value corresponding to the respective green picture element using a calculated value derived from values of a first kernel, the first kernel comprising a plurality of green picture elements of the first green color channel that are selected based on a position of the respective green picture element, to generate transformed second green color channel data;

for each respective red picture element of a plurality of red picture elements of the red color channel, modify an initial value corresponding to the respective red picture element using a calculated value derived from values of a second kernel, the second kernel comprising a plurality of green picture elements of one or more of the first green color channel and the second green color channel that are selected based on a position of the respective red picture element, to generate transformed red color channel data;

for each respective blue picture element of a plurality of blue picture elements of the blue color channel, modify an initial value corresponding to the respective blue picture element using a calculated value derived from values of a third kernel, the third kernel comprising a plurality of green picture elements of one or more of the first green color channel and the second green color channel that are selected based on a position of the respective blue picture element, to generate transformed blue color channel data; and

compress the transformed second green color channel data, data corresponding to the first green color channel, the transformed red color channel data, and the transformed blue color channel data, to generate compressed mosaiced color motion video image data.

17. The portable electronic device of claim 16, wherein the one or more hardware processors are further configured to, for each respective green picture element of the plurality of green picture elements of the second green color channel, subtract the calculated value from the initial value corresponding to the respective green picture element, and

wherein, for each respective green picture element of the plurality of green picture elements of the second green color channel, the calculated value comprises an average of the values of the plurality of green picture elements in the first kernel, and the plurality of green picture elements in the first kernel comprise at least four picture elements which are on opposite sides of the respective green picture element.

18. The portable electronic device of claim 17 further comprising:

a lens mount; and

a removable lens attachable to the lens mount.

19. The portable electronic device of claim 16, wherein the compression comprises application of a lossy compression algorithm, and wherein the one or more hardware processors are further configured to, for the plurality of image frames, apply a non-linear pre-emphasis function to the mosaiced color motion video image data prior to the compression.

20. The portable electronic device of claim 19, wherein the one or more hardware processors are configured to apply the pre-emphasis function to at least the transformed red color channel data, the transformed blue color channel data, and the transformed second green color channel data.

21. A system configured to decode color motion video image data, the system comprising:

one or more image sensors; and

one or more hardware processors configured to;

access encoded color image data, wherein;

the encoded color image data was generated from mosaiced color motion video image data initially acquired by the one or more image sensors,

the encoded color image data includes at least a first green color channel, a second green color channel, a red color channel, and a blue color channel, and

the encoded color image data was encoded at least partly by, for a plurality of motion video image frames;

for each respective green picture element of a plurality of green picture elements of the second green color channel, modifying an initial value corresponding to the respective green picture element using a calculated value derived from values of a first kernel, the first kernel comprising a plurality of green picture elements of the first green color channel of the plurality of color channels that are selected based on a position of the respective green picture element, to generate transformed second green color channel data,

for each respective red picture element of a plurality of red picture elements of the red color channel, modifying an initial value corresponding to the respective red picture element using a calculated value derived from values of a second kernel, the second kernel comprising a plurality of green picture elements of one or more of the first green color channel and the second green color channel that are selected based on a position of the respective red picture element, to generate transformed red color channel data,

for each respective blue picture element of a plurality of blue picture elements of the blue color channel, modifying an initial value corresponding to the respective blue picture element using a calculated value derived from values of a third kernel, the third kernel comprising a plurality of green picture elements of one or more of the first green color channel and the second green color channel that are selected based on a position of the respective blue picture element, to generate transformed blue color channel data, and

compressing the transformed second green color channel data, data corresponding to the first green color channel, the transformed red color channel data, and the transformed blue color channel data, to generate compressed mosaiced color motion video image data; and

decode the accessed encoded color motion video image data to reproduce the red color channel that has been transformed, the blue color channel that has been transformed, data corresponding to the first green color channel, and the second green color channel that has been transformed.

22. The system of claim 21 wherein the system comprises a portable electronic device having imaging capability, and the system further comprises:

a lens mount; and

a removable lens attachable to the lens mount.