Compression and decoding of single sensor color image data

• US 9,438,799 B2
• Filed: 04/01/2015
• Issued: 09/06/2016
• Est. Priority Date: 03/22/2006
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A.
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US14/676,776
• Publication Number:
  US 20150281579A1
• Time in Patent Office: 524 Days
• Field of Search: 382/232
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: H04N 1/648 Transmitting or storing the primary (additive or subtractive) colour signals; Compression thereof H04N1/642 - H04N1/646 take precedence
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A method is described to greatly improve the efficiency of and reduce the complexity of image compression when using single-sensor color imagers for video acquisition. The method in addition allows for this new image compression type to be compatible with existing video processing tools, improving the workflow for film and television production.

Claims

1. A method for previewing captured image data, the method comprising:

capturing, by an image sensor of a camera, image data, the image data comprising a plurality of image planes;

encoding, by an image processor, the captured image data into a plurality of encoded image planes, each encoded image plane representative of one or more of the image planes of the image data, such that all of the encoded image planes form an image at a first resolution and a subset of less than all of the encoded image planes form an image at a second resolution less than the first resolution; and

providing, to an external display, the subset of the encoded image planes for a substantially real-time preview of the image at the second resolution, the external display configured to decode the subset of encoded image planes to obtain the image at the second resolution.

2. The method of claim 1, wherein the second resolution comprises a resolution that is approximately one-quarter the resolution of the first resolution.

3. The method of claim 1, wherein the plurality of encoded image planes comprises a first encoded image plane representative of a first color and a second encoded image plane representative of the first color, and wherein the subset of encoded image planes comprises only one of the first encoded image plane and the second encoded image plane.

4. The method of claim 1, wherein decoding the subset of encoded image planes comprises:

decoding one or more of the subset of encoded image planes to produce a subset of the plurality of image planes comprising less than all of the plurality of image planes; and

combining the subset of the plurality of image planes to produce the image at the second resolution.

5. The method of claim 1, wherein each pixel of the image at the second resolution comprises a red sub-pixel, a blue sub-pixel, and a green sub-pixel.

6. The method of claim 1, wherein each pixel of the image at the first resolution comprises a red sub-pixel, a blue sub-pixel, a first green sub-pixel, and a second green sub-pixel.

7. The method of claim 1, wherein the external display is configured to display the image at the second resolution.

8. The method of claim 1, further comprising:

in response to the preview of the image at the second resolution, receiving an editing instruction from a user.

17. The method of claim 1, wherein the image data comprises a set of pixels, and wherein each of the plurality of image planes comprises a subset of the set of pixels.

18. The method of claim 17, wherein each of one or more of the plurality of image planes comprises approximately one quarter of the pixels in the set of pixels.

19. The method of claim 17, wherein each of one or more of the plurality of image planes comprises approximately one half of the pixels in the set of pixels.

20. The method of claim 17, wherein each of one or more of the plurality of image planes comprises pixels corresponding to one color.

21. The method of claim 17, wherein each of one or more of the plurality of image planes comprises pixels corresponding to two or more colors.

22. The method of claim 1, wherein the image data includes metadata describing camera settings for the camera.

23. The method of claim 8, wherein the received editing instruction is applied to the image at the first resolution.

24. The method of claim 1, wherein decoding the subset of encoded image planes to obtain the image at the second resolution comprises bypassing de-Bayer filtering.

25. The method of claim 1, wherein the second resolution comprises a resolution that is less than half the resolution of the first resolution.

26. The method of claim 1, wherein the image data comprises video image data.

27. The method of claim 26, wherein the plurality of encoded image planes comprises one or more first encoded image planes and one or more second encoded image planes, wherein each of the first encoded image planes includes more image data than each of the second encoded image planes.

28. The method of claim 27, wherein the subset of encoded image planes comprises only one of the one or more first encoded image planes.

29. The method of claim 27, wherein the subset of encoded image planes comprises only one of the one or more second encoded image planes.

30. The method of claim 1, wherein the external display is configured to display the image at the second resolution, and further comprising:

in response to the preview of the image at the second resolution, receiving an editing instruction from a user.

31. The method of claim 30, wherein the received editing instruction is applied to the image at the first resolution.

32. The method of claim 30, wherein the image data includes metadata describing camera settings for the camera.

9. A system for previewing captured image data, the system comprising:

an image sensor configured to capture image data, the image data comprising a plurality of image planes;

an image processor configured to encode the captured image data into a plurality of encoded image planes, each encoded image plane representative of one or more of the image planes of the image data, such that all of the encoded image planes form an image at a first resolution and a subset of less than all of the encoded image planes form an image at a second resolution less than the first resolution;

an output to provide, to an external display, the subset of the encoded image planes for a substantially real-time preview of the image at the second resolution, the external display configured to decode the subset of encoded image planes to obtain the image at the second resolution.

10. The system of claim 9, wherein the second resolution comprises a resolution that is approximately one-quarter the resolution of the first resolution.

11. The system of claim 9, wherein the plurality of encoded image planes comprises a first encoded image plane representative of a first color and a second encoded image plane representative of the first color, and wherein the subset of encoded image planes comprises only one of the first encoded image plane and the second encoded image plane.

12. The system of claim 9, wherein decoding the subset of encoded image planes comprises:

decoding one or more of the subset of encoded image planes to produce a subset of the plurality of image planes comprising less than all of the plurality of image planes; and

combining the subset of the plurality of image planes to produce the image at the second resolution.

13. The system of claim 9, wherein each pixel of the image at the second resolution comprises a red sub-pixel, a blue sub-pixel, and a green sub-pixel.

14. The system of claim 9, wherein each pixel of the image at the first resolution comprises a red sub-pixel, a blue sub-pixel, a first green sub-pixel, and a second green sub-pixel.

15. The system of claim 9, wherein the external display is configured to display the image at the second resolution.

16. The system of claim 9, further comprising:

an input configured to, in response to the preview of the image at the second resolution, receive an editing instruction from a user.

33. The system of claim 9, wherein the image data comprises a set of pixels, and wherein each of the plurality of image planes comprises a subset of the set of pixels.

34. The system of claim 33, wherein each of one or more of the plurality of image planes comprises approximately one quarter of the pixels in the set of pixels.

35. The system of claim 33, wherein each of one or more of the plurality of image planes comprises approximately one half of the pixels in the set of pixels.

36. The system of claim 33, wherein each of one or more of the plurality of image planes comprises pixels corresponding to one color.

37. The system of claim 33, wherein each of one or more of the plurality of image planes comprises pixels corresponding to two or more colors.

38. The system of claim 9, wherein the image data includes metadata describing settings for the system.

39. The system of claim 16, wherein the received editing instruction is applied to the image at the first resolution.

40. The system of claim 9, wherein decoding the subset of encoded image planes to obtain the image at the second resolution comprises bypassing de-Bayer filtering.

41. The system of claim 9, wherein the second resolution comprises a resolution that is less than half the resolution of the first resolution.

42. The system of claim 9, wherein the image data comprises video image data.

43. The system of claim 42, wherein the plurality of encoded image planes comprises one or more first encoded image planes and one or more second encoded image planes, wherein each of the first encoded image planes includes more image data than each of the second encoded image planes.

44. The system of claim 43, wherein the subset of encoded image planes comprises only one of the one or more first encoded image planes.

45. The system of claim 43, wherein the subset of encoded image planes comprises only one of the one or more second encoded image planes.

46. The system of claim 9, wherein the external display is configured to display the image at the second resolution, and further comprising:

an input configured to, in response to the preview of the image at the second resolution, receive an editing instruction from a user.

47. The system of claim 46, wherein the received editing instruction is applied to the image at the first resolution.

48. The system of claim 46, wherein the image data includes metadata describing camera settings for the camera.

49. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a hardware processor, cause a camera system to perform steps comprising:

capturing, by an image sensor of a camera system, image data, the image data comprising a plurality of image planes;

encoding, by an image processor of a camera system, the captured image data into a plurality of encoded image planes, each encoded image plane representative of one or more of the image planes of the image data, such that all of the encoded image planes form an image at a first resolution and a subset of less than all of the encoded image planes form an image at a second resolution less than the first resolution; and

providing, to an external display, the subset of the encoded image planes for a substantially real-time preview of the image at the second resolution, the external display configured to decode the subset of encoded image planes to obtain the image at the second resolution.

50. The non-transitory computer-readable storage medium of claim 49, wherein the second resolution comprises a resolution that is approximately one-quarter the resolution of the first resolution.

51. The non-transitory computer-readable storage medium of claim 49, wherein the plurality of encoded image planes comprises a first encoded image plane representative of a first color and a second encoded image plane representative of the first color, and wherein the subset of encoded image planes comprises only one of the first encoded image plane and the second encoded image plane.

52. The non-transitory computer-readable storage medium of claim 49, wherein decoding the subset of encoded image planes comprises:

decoding one or more of the subset of encoded image planes to produce a subset of the plurality of image planes comprising less than all of the plurality of image planes; and

combining the subset of the plurality of image planes to produce the image at the second resolution.

53. The non-transitory computer-readable storage medium of claim 49, wherein each pixel of the image at the second resolution comprises a red sub-pixel, a blue sub-pixel, and a green sub-pixel.

54. The non-transitory computer-readable storage medium of claim 49, wherein each pixel of the image at the first resolution comprises a red sub-pixel, a blue sub-pixel, a first green sub-pixel, and a second green sub-pixel.

55. The non-transitory computer-readable storage medium of claim 49, wherein the external display is configured to display the image at the second resolution.

56. The non-transitory computer-readable storage medium of claim 49, wherein the instructions, when executed by the hardware processor, cause the camera system to perform further steps comprising:

in response to the preview of the image at the second resolution, receiving an editing instruction from a user.

57. The non-transitory computer-readable storage medium of claim 49, wherein the image data comprises a set of pixels, and wherein each of the plurality of image planes comprises a subset of the set of pixels.

58. The non-transitory computer-readable storage medium of claim 57, wherein each of one or more of the plurality of image planes comprises approximately one quarter of the pixels in the set of pixels.

59. The non-transitory computer-readable storage medium of claim 57, wherein each of one or more of the plurality of image planes comprises approximately one half of the pixels in the set of pixels.

60. The non-transitory computer-readable storage medium of claim 57, wherein each of one or more of the plurality of image planes comprises pixels corresponding to one color.

61. The non-transitory computer-readable storage medium of claim 57, wherein each of one or more of the plurality of image planes comprises pixels corresponding to two or more colors.

62. The non-transitory computer-readable storage medium of claim 49, wherein the image data includes metadata describing camera settings for the camera system.

63. The non-transitory computer-readable storage medium of claim 62, wherein the received editing instruction is applied to the image at the first resolution.

64. The non-transitory computer-readable storage medium of claim 49, wherein decoding the subset of encoded image planes to obtain the image at the second resolution comprises bypassing de-Bayer filtering.

65. The non-transitory computer-readable storage medium of claim 49, wherein the second resolution comprises a resolution that is less than half the resolution of the first resolution.

66. The non-transitory computer-readable storage medium of claim 49, wherein the image data comprises video image data.

67. The non-transitory computer-readable storage medium of claim 66, wherein the plurality of encoded image planes comprises one or more first encoded image planes and one or more second encoded image planes, wherein each of the first encoded image planes includes more image data than each of the second encoded image planes.

68. The non-transitory computer-readable storage medium of claim 67, wherein the subset of encoded image planes comprises only one of the one or more first encoded image planes.

69. The non-transitory computer-readable storage medium of claim 67, wherein the subset of encoded image planes comprises only one of the one or more second encoded image planes.

70. The non-transitory computer-readable storage medium of claim 49, wherein the external display is configured to display the image at the second resolution, and wherein the instructions, when executed by the hardware processor, cause the camera system to perform further steps comprising:

in response to the preview of the image at the second resolution, receiving an editing instruction from a user.

71. The non-transitory computer-readable storage medium of claim 70, wherein the received editing instruction is applied to the image at the first resolution.

72. The non-transitory computer-readable storage medium of claim 70, wherein the image data includes metadata describing camera settings for the camera.