Layered scene decomposition CODEC system and methods

• US 20190068973A1
• Filed: 08/23/2017
• Published: 02/28/2019
• Est. Priority Date: 08/23/2017
• Status: Active Grant

• Current Assignee: Avalon Holographics Inc.
• Original Assignee: Avalon Holographics Inc.
• Inventors: Hamilton, Matthew, Rumbolt, Chuck, Benoit, Donovan, Troke, Matthew, Lockyer, Robert
• Granted Patent: US 10,432,944 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/184 the unit being bits, e.g. of the compressed video stream
• CPC Class Codes: H04N 19/25 with scene description coding, e.g. binary format for scenes [BIFS] compression
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A system and methods for a CODEC driving a real-time light field display for multi-dimensional video streaming, interactive gaming and other light field display applications is provided applying a layered scene decomposition strategy. Multi-dimensional scene data is divided into a plurality of data layers of increasing depths as the distance between a given layer and the plane of the display increases. Data layers are sampled using a plenoptic sampling scheme and rendered using hybrid rendering, such as perspective and oblique rendering, to encode light fields corresponding to each data layer. The resulting compressed, (layered) core representation of the multi-dimensional scene data is produced at predictable rates, reconstructed and merged at the light field display in real-time by applying view synthesis protocols, including edge adaptive interpolation, to reconstruct pixel arrays in stages (e.g. columns then rows) from reference elemental images.

Claims

1. A computer-implemented method comprising the steps of:

i. receiving from a source a data set comprising light field data or a 3D description of a scene;

ii. partitioning said data set into a plurality of data layers, each data layer comprising elemental image content; and

iii. sampling and rendering each data layer of the plurality of data layers to encode a light field corresponding to each data layer and produce a set of compressed data encoding a set of light fields.

2. The computer-implemented method according to claim 1, wherein the partitioning of the data set into a plurality of data layers is implemented according to a layering scheme to facilitate the encoding of the set of light fields as a distribution of light fields arranged such that the depth of each light field increases as the distance between said light field and a display surface increases.

3. The computer implemented method according to claim 2, wherein the depth of each light field increases exponentially.

4. The computer-implemented method according to claim 1, wherein the sampling of each data layer of said plurality of data layers is implemented according to one or more plenoptic sampling schemes specifying, for each light field corresponding to each data layer, reference elemental images sampled from the elemental image content of each data layer.

5. The computer-implemented method according to claim 1, wherein the rendering of a data layer is executed by applying a rendering technique selected to efficiently encode the light field corresponding to said data layer.

6. The computer-implemented method according to claim 5, wherein the rendering technique selected is perspective rendering, or oblique rendering based on which technique would use the least number of draw calls.

7. The computer-implemented method according to claim 6, where the rendering technique selected is ray casting.

8. The computer-implemented method according to claim 1, further comprising the step of assigning a variable fractional bit representation to each data layer of the plurality of data layers to optimize compression of the data set.

9. The computer-implemented method according to claim 1, further comprising the steps of transmitting and decoding the set of compressed data to construct the set of light fields.

10. The computer-implemented method according to claim 9, wherein the step of transmitting the compressed data is performed using a 2D display cable.

11. The computer-implemented method according to claim 9, wherein the step of decoding the set of compressed data is implemented by executing a view synthesis protocol for each data layer sampled and rendered to encode the light field corresponding to each data layer.

12. The computer-implemented method according to claim 11, wherein each view synthesis protocol comprises instructions for the multi-stage reconstruction of a pixel array from reference elemental images used to construct the light field corresponding to each data layer.

13. The computer-implemented method according to claim 12, wherein the multi-stage reconstruction of the pixel array is performed in a first stage by column decoding and in a second stage by row decoding.

14. The computer-implemented method according to claim 12, wherein the multi-stage reconstruction of the pixel array is performed in a first stage by row decoding and in a second stage by column decoding.

15. The computer-implemented method according to claim 11, wherein one or more of the view synthesis protocols is executed with the application of edge adaptive interpolation to optimize the reconstruction of the pixel array.

16. The computer-implemented method according to claim 9, further comprising the step of merging the light fields in the set of light fields to produce a display light field at a display.

17. The computer-implemented method according to claim 16, wherein the set of compressed data is used to produce a core encoded representation of the display light field.

18. The computer-implemented method according to claim 16, further comprising the steps of encoding and compressing a residue encoded representation of the display light field.

19. The computer-implemented method according to claim 18, wherein the core and residue encoded representations of the set of light fields are used to produce the display light field.

20. The computer-implemented method according to claim 16, wherein the display light field represents a frustum volume of the display.

21. The computer-implemented method according to claim 16, wherein the display light field represents the inner frustum volume and outer frustum volume of the display.

22. A computer-implemented system, comprising:

a source for providing a data set, the data set comprising light field data or a 3D description of a scene;

an encoder in communication with said source, comprising one or more processors configured to;

i. partition the data set received from said source into a plurality of data layers, each data layer comprising elemental image content; and

ii. sample and render each data layer of the plurality of data layers to encode a light field corresponding to each data layer and produce a set of compressed data encoding a set of light fields;

a means for transmitting the set of compressed data from the encoder to a decoder; and

a decoder comprising one or more processors configured to decode the set of compressed data to construct a set of light fields.

23. The computer-implemented system according to claim 22, wherein the one or more processors of the encoder partition the data set into a plurality of data layers by implementing a layering scheme to facilitate the encoding of the set of light fields as a distribution of light fields arranged such that the depth of each light field increases as the distance between said light field and a display surface increases.

24. The computer-implemented system according to claim 23, wherein the depth of each light field increases exponentially.

25. The computer-implemented system according to claim 22, wherein the one or more processors of the encoder sample each data layer of said plurality of data layers by implementing one or more plenoptic sampling schemes specifying, for each light field corresponding to each data layer, reference elemental images sampled from the elemental image content of each data layer.

26. The computer-implemented system according to claim 22, wherein the one or more processors of the encoder render each data layer by executing a rendering technique selected to efficiently encode the light field corresponding to said data layer.

27. The computer-implemented system according to claim 26, wherein the rendering technique selected is perspective rendering, or oblique rendering based on which technique would use the least number of draw calls.

28. The computer-implemented system according to claim 27, wherein the rendering technique selected is ray casting.

29. The computer-implemented system according to claim 22, wherein the one or more processors of the encoder are further configured to assign a variable fractional bit representation to each data layer of the plurality of data layers to optimize compression of the data set.

30. The computer-implemented system according to claim 22, wherein the transmission means is a 2D display cable.

31. The computer-implemented system according to claim 22 wherein the one or more processors of the decoder implement the decoding of the set of compressed data by executing a view synthesis protocol for each data layer sampled and rendered to encode the light field corresponding to each data layer.

32. The computer-implemented system according to claim 31, wherein each view synthesis protocol comprises instructions for the multi-stage reconstruction of a pixel array from reference elemental images used to construct the light field corresponding to each data layer.

33. The computer-implemented system according to claim 32, wherein the multi-stage reconstruction of the pixel array is performed in a first stage by column decoding and in a second stage by row decoding.

34. The computer-implemented system according to claim 32, wherein the multi-stage reconstruction of the pixel array is performed in a first stage by row decoding and in a second stage by column decoding.

35. The computer-implemented system according to claim 31, wherein one or more of the view synthesis protocols is executed with the application of edge adaptive interpolation to optimize the reconstruction of the pixel array.

36. The computer-implemented system according to claim 22, wherein the one or more processors of the decoder are further configured to merge the light fields in the set of light fields to produce a display light field at a display.

37. The computer-implemented system according to claim 36, wherein the set of compressed data is used to produce a core encoded representation of the display light field.

38. The computer-implemented system according to claim 36, wherein the one or more processors of the encoder are further configured to encode and compress a residue encoded representation of the display light field.

39. The computer-implemented system according to claim 38, wherein the one or more processors of the decoder use the core and residue encoded representations of the set of light fields to produce the display light field.

40. The computer-implemented system according to claim 36, wherein the display light field represents a frustum volume of the display.

41. The computer-implemented system according to claim 36, wherein the display light field represents an inner frustum volume and outer frustum volume of the display.

42. A computer-implemented system comprising one or more processors and one or more memory components comprising instructions configured to cause the one or more processors to perform a method comprising the steps of:

i. receiving from a source a data set comprising light field data or a 3D description of a scene;

ii. partitioning said data set into a plurality of data layers, each data layer comprising elemental image content; and

iii. sampling and rendering each data layer of the plurality of data layers to encode a light field corresponding to each data layer and produce a set of compressed data encoding a set of light fields.

43. The computer-implemented system according to claim 42, wherein the method step of partitioning the data set into a plurality of data layers is implemented by a layering scheme to facilitate the encoding of the set of light fields as a distribution of light fields arranged such that the depth of each light field increases as the distance between said light field and a display surface increases.

44. The computer implemented system according to claim 43, wherein the depth of each light field increases exponentially.

45. The computer-implemented system according to claim 42, wherein the method step of sampling each data layer of said plurality of data layers is implemented by one or more plenoptic sampling schemes specifying, for each light field corresponding to each data layer, reference elemental images sampled from the elemental image content of each data layer.

46. The computer-implemented system according to claim 42, wherein the method step of rendering each data layer is executed by applying a rendering technique selected to efficiently encode the light field corresponding to said data layer.

47. The computer-implemented system according to claim 46, wherein the rendering technique selected is perspective rendering, or oblique rendering based on which technique would use the least number of draw calls.

48. The computer-implemented system according to claim 47, wherein the rendering technique selected is ray casting.

49. The computer-implemented system according to claim 42, wherein the method further comprises the step of assigning a variable fractional bit representation to each data layer of the plurality of data layers to optimize compression of the data set.

50. The computer-implemented system according to claim 42, wherein the method further comprises the steps of transmitting and decoding the set of compressed data to construct the set of light fields.

51. The computer-implemented system according to claim 50, wherein the method step of transmitting the compressed data is performed using a 2D display cable.

52. The computer-implemented system according to claim 50, wherein the method step of decoding the set of compressed data is implemented by executing a view synthesis protocol for each data layer sampled and rendered to encode the light field corresponding to each data layer.

53. The computer-implemented system according to claim 52, wherein each view synthesis protocol comprises instructions for the multi-stage reconstruction of a pixel array from reference elemental images used to construct the light field corresponding to each data layer.

54. The computer-implemented system according to claim 53, wherein the multi-stage reconstruction of the pixel array is performed in a first stage by column decoding and in a second stage by row decoding.

55. The computer-implemented system according to claim 53, wherein the multi-stage reconstruction of the pixel array is performed in a first stage by row decoding and in a second stage by column decoding.

56. The computer-implemented system according to claim 52, wherein one or more of the view synthesis protocols is executed with the application of edge adaptive interpolation to optimize the reconstruction of the pixel array.

57. The computer-implemented system according to claim 50, wherein the method further comprises the step of merging the light fields in the set of light fields to produce a display light field at a display.

58. The computer-implemented system according to claim 47, wherein the set of compressed data is used to produce a core encoded representation of the display light field.

59. The computer-implemented system according to claim 47, wherein the method further comprises the steps of encoding and compressing a residue encoded representation of the display light field.

60. The computer-implemented system according to claim 49, wherein the core and residue encoded representations of the set of light fields are used to produce the display light field.

61. The computer-implemented system according to claim 47, wherein the display light field represents a frustum volume of the display.

62. The computer-implemented system according to claim 47, wherein the display light field represents an inner frustum volume and outer frustum volume of the display.

63. A non-transitory, computer readable medium comprising instructions configured to cause the one or more processors to perform a method comprising the steps of:

i. receiving from a source a data set comprising light field data or a 3D description of a scene;

ii. partitioning said data set into a plurality of data layers, each data layer comprising elemental image content; and

iii. sampling and rendering each data layer of the plurality of data layers to encode a light field corresponding to each data layer and produce a set of compressed data encoding a set of light fields.

64. The non-transitory, computer readable medium according to claim 63, wherein the method step of partitioning the data set into a plurality of data layers is implemented by a layering scheme to facilitate the encoding of the set of light fields as a distribution of light fields arranged such that the depth of each light field increases as the distance between said light field and a display surface increases.

65. The non-transitory, computer readable medium according to claim 64, wherein the depth of each light field increases exponentially.

66. The non-transitory, computer readable medium according to claim 63, wherein the method step of sampling each data layer of said plurality of data layers is implemented by one or more plenoptic sampling schemes specifying, for each light field corresponding to each data layer, reference elemental images sampled from the elemental image content of each data layer.

67. The non-transitory, computer readable medium according to claim 63, wherein the method step of rendering each data layer is executed by applying a rendering technique selected to efficiently encode the light field corresponding to said data layer.

68. The non-transitory, computer readable medium according to claim 67, wherein the rendering technique selected is perspective rendering, or oblique rendering based on which technique would use the least number of draw calls.

69. The non-transitory, computer readable medium according to claim 68, wherein the rendering technique selected is ray casting.

70. The non-transitory, computer readable medium according to claim 63, wherein the method further comprises the step of assigning a variable fractional bit representation to each data layer of the plurality of data layers to optimize compression of the data set.

71. The non-transitory, computer readable medium according to claim 63, wherein the method further comprises the steps of transmitting and decoding the set of compressed data to construct the set of light fields.

72. The non-transitory, computer readable medium according to claim 71, wherein the method step of transmitting the compressed data is performed using a 2D display cable.

73. The non-transitory, computer readable medium according to claim 71, wherein the method step of decoding the set of compressed data is implemented by executing a view synthesis protocol for each data layer sampled and rendered to encode the light field corresponding to each data layer.

74. The non-transitory, computer readable medium according to claim 73, wherein each view synthesis protocol comprises instructions for the multi-stage reconstruction of a pixel array from reference elemental images used to construct the light field corresponding to each data layer.

75. The non-transitory, computer readable medium according to claim 74, wherein the multi-stage reconstruction of the pixel array is performed in a first stage by column decoding and in a second stage by row decoding.

76. The non-transitory, computer readable medium according to claim 74, wherein the multi-stage reconstruction of the pixel array is performed in a first stage by row decoding and in a second stage by column decoding.

77. The non-transitory, computer readable medium according to claim 73, wherein one or more of the view synthesis protocols is executed with the application of edge adaptive interpolation to optimize the reconstruction of the pixel array.

78. The non-transitory, computer readable medium according to claim 71, wherein the method further comprises the step of merging the light fields in the set of light fields to produce a display light field at a display.

79. The non-transitory, computer readable medium according to claim 78, wherein the set of compressed data is used to produce a core encoded representation of the display light field.

80. The non-transitory, computer readable medium according to claim 78, wherein the method further comprises the steps of encoding and compressing a residue encoded representation of the display light field.

81. The non-transitory, computer readable medium according to claim 80, wherein the core and residue encoded representations of the set of light fields are used to produce the display light field.

82. The non-transitory, computer readable medium according to claim 78, wherein the display light field represents a frustum volume of the display.

83. The non-transitory, computer readable medium according to claim 78, wherein the display light field represents an inner frustum volume and an outer frustum volume of the display.