Methods for Full Parallax Compressed Light Field 3D Imaging Systems

• US 20150201176A1
• Filed: 01/07/2015
• Published: 07/16/2015
• Est. Priority Date: 01/10/2014
• Status: Active Grant

• Current Assignee: Ostendo Technologies Inc.
• Original Assignee: Ostendo Technologies Inc.
• Inventors: Alpaslan, Zahir Y., El-Ghoroury, Hussein S., Graziosi, Danillo B.
• Granted Patent: US 10,244,223 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/172 image signals comprising non-image signal components, e.g. headers or format information
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0074 Stereoscopic image analysis

Abstract

A compressed light field imaging system is described. The light field 3D data is analyzed to determine optimal subset of light field samples to be (acquired) rendered, while the remaining samples are generated using multi-reference depth-image based rendering. The light field is encoded and transmitted to the display. The 3D display directly reconstructs the light field and avoids data expansion that usually occurs in conventional imaging systems. The present invention enables the realization of full parallax 3D compressed imaging system that achieves high compression performance while minimizing memory and computational requirements.

Claims

1. A method for light field imaging systems comprising:

compressed capturing of light field data from a light field to provide compressed light field data; and

reconstructing and displaying the light field from the compressed light field data at a light field display system.

2. The method of claim 1, wherein:

the compressed capturing of the light field data includes compressed rendering of the light field data, and display-matched encoding of the compressed rendered light field data that matches capabilities of the light field display system; and

reconstruction of the light field directly at the light field display system without first decompressing the compressed light field data before the light field is transmitted to the light field display system.

3. The method of claim 1, further comprising rendering computer-generated images to obtain texture and/or depth of selected views of a scene.

4. The method of claim 1, wherein the light field data is captured using multiple capture methods or cameras to obtain texture and/or depth of selected views of a scene.

5. The method of claim 1, wherein computer-generated images are used along with one or more real-world captured light fields, in an augmented reality application.

6. The method of claim 1, further comprising transmitting the compressed light field data in a bitstream format to a display capable of displaying the light field for observing a 3D image.

7. The method of claim 1, further comprising transmitting the compressed light field data in a bitstream format to a display capable of displaying the light field to present a 3D image viewable without the use of glasses.

8. The method of claim 1, further comprising transmitting the compressed light field data in a bitstream format to a display capable of displaying the light field without further processing.

9. The method of claim 1, further comprising storing the compressed light field data in a bitstream format in a storage media prior to reconstructing and displaying the light field.

10. The method of claim 1, wherein the compressed capturing further includes analysis of 3D scene data to select reference hogels representing the 3D scene.

11. The method of claim 10, wherein the compressed capturing further includes creating disparity maps for the reference hogels and the synthesis of target hogels from the reference hogels texture and disparity maps.

12. The method of claim 11 wherein the target hogels are synthesized using multiple-reference depth-image based rendering.

13. The method of claim 12, further comprising selecting multiple reference hogels.

14. The method of claim 13, further comprising forward warping of the disparity maps of the selected reference hogels.

15. The method of claim 14, further comprising filtering of the forward warped disparity maps.

16. The method of claim 15, further comprising merging the forward warped disparity maps into a single disparity map.

17. The method of claim 16, further comprising backward warping of textures according to the merged disparity map.

18. The method of claim 17, wherein fractional pixel shifting is used in the backward warping of textures.

19. The method of claim 17, further comprising capturing and using reference hogels in the compressed capturing using a higher resolution than the resolution of the light field display system.

20. The method of claim 17, further comprising hole filling after the backward warping.

21. The method of claim 10, wherein the compressed capturing comprises compressed rendering and display-matched compression to form the compressed light field data, and wherein analysis of 3D scene data to select reference hogels representing the 3D scene comprises a visibility test to choose the reference hogels to be rendered in the compressed rendering.

22. The method of claim 21, further comprising selecting a subset of reference hogels from a larger set of reference hogels.

23. The method of claim 21, further comprising selecting an initial set of reference hogels, and adding more reference hogels to better represent objects of the 3D scene by frusta of the initial set of reference hogels.

24. The method of claim 21, further comprising preprocessing 3D scene data before the visibility test to extract information from the 3D scene data for the visibility test.

25. The method of claim 24 wherein the preprocessing utilizes computer generated scenes captured by computer graphics methods.

26. The method of claim 24 further comprising preprocessing utilizing real world or live scenes captured by one or more cameras or camera types.

27. The method of claim 10, including rendering of the reference hogels.

28. The method of claim 27, further comprising low-pass filtering of the rendered reference hogels to avoid non-resolvable features.

29. The method of claim 27, further comprising obtaining per-pixel depth of the reference hogels.

30. The method of claim 29, further comprising converting per-pixel depth to disparity and quantizing the disparity information during the depth to disparity conversion after reference hogel rendering.

31. The method of claim 1, further comprising formatting the compressed light field data by an encoder to generate a bitstream for transmission to the light field display system for decoding, and the display of the light field.

32. The method of claim 31, wherein the bitstream is matched to the light field display system.

33. The method of claim 32, further comprising dividing the light field into N×

N blocks of hogels, each for independent encoding during the compressed capturing and for independent decoding at the light field display system.

34. The method of claim 33, further comprising selecting one or more seed hogels for each of the N×

N blocks of hogels for encoding, and encoding residual hogels relative to the seed hogels.

35. The method of claim 34, wherein the encoding comprises texture encoding of seed and residual hogels.

36. The method of claim 35, further comprising synthesizing a prediction for the residual hogels, using a texture and a disparity map of a plurality of seed hogels.

37. The method of claim 34, wherein the encoding comprises disparity encoding of seed and residual hogels.

38. The method of claim 34, further comprising coding the seed and residual hogels with a bit allocation algorithm.

39. The method of claim 38, further comprising decoding the seed and residual hogels at the light field display system using a decoding algorithm, the decoding algorithm being complimentary to the bit allocation algorithm.

40. The method of claim 39, further comprising parallel decoding of all seed and residual hogels at the light field display system with multiprocessing techniques using multiple decoding units.

41. The method of claim 40, further comprising packetizing the bit allocation for distribution among multiple decoding units.

42. The method of claim 1 further comprising performing a hierarchical compressed capturing of the light field data, and performing a hierarchical decompression at the light field display system.

43. The method of claim 1 further comprising the compressed capturing of a dynamic light field with temporal correlation.

44. The method of claim 43 further comprising using motion estimation in the compressed capturing of the light field data.

45. The method of claim 44 wherein the compressed capturing uses depth or disparity image-based rendering, and further comprising reusing hardware and/or software that is used for the depth or disparity image-based rendering to perform both motion estimation and motion compensation.

46. The method of claim 1 further comprising time multiplexing the compressed light field data by rotations and translations of a light field emitter of the light field display system.

47. The method of claim 46 further comprising temporally multiplexing the compressed light field data into smaller subsets organized spatially or angularly.

48. The method of claim 47 further comprising creating subsets that have a predetermined amount of correlation.

49. The method of claim 46 further comprising reusing display pixels, memory and compressed light field data by time multiplexed reconstruction of the compressed light field data, thereby increasing the field of view of the light field display system.

50. The method of claim 46 wherein the time multiplexing further comprises creating multiplexing segments that naturally divide the compressed light field data into hogel modulation groups.

51. The method of claim 50 further comprising adaptive allocation of the interface bandwidth, and wherein the time multiplexing of the display-matched encoder'"'"'s compression algorithm together with the adaptive allocation of the interface bandwidth.

52. The method of claim 46 further comprising dividing the compressed light field data into hogel groups that are used for parallel implementation of compressed rendering and display-matched encoding.

53. A method for light field imaging systems comprising:

compressed capturing of light field data from a light field to provide compressed light field data;

the compressed capturing of the light field data including compressed rendering of the light field data and display-matched encoding of the rendered light field data that matches capabilities of a light field display system, the compressed capturing including analysis of 3D scene data to select reference hogels representing the 3D scene; and

reconstructing and displaying the display-matched encoding of the rendered light field data at the light field display system.

54. The method of claim 53, wherein the compressed capturing further includes synthesis of target hogels from the reference hogels.

55. The method of claim 54 wherein the target hogels are synthesized using multiple-reference depth-image based rendering.

56. The method of claim 53, including rendering of the reference hogels.

57. The method of claim 56 further comprising obtaining per-pixel depth of the reference hogels.

58. The method of claim 57, further comprising converting per-pixel depth to disparity and quantizing the disparity during the depth to disparity conversion after reference hogel rendering.

59. The method of claim 53, wherein the selected reference hogels are captured with a higher resolution than the resolution of the light field display system.

60. The method of claim 53, further comprising dividing the light field data into N×

N blocks of hogels, each for independent encoding during the compressed capturing and for independent decoding at the light field display system.

61. The method of claim 60, further comprising selecting one or more seed hogels for each of the N×

N blocks of hogels for encoding, and encoding residual hogels relative to the seed hogels.

62. The method of claim 61, wherein the encoding comprises texture encoding of seed and residual hogels.

63. The method of claim 61, wherein the encoding comprises disparity encoding of seed and residual hogels.

64. The method of claim 63, further comprising parallel decoding of all seed and residual hogels at the light field display system using multiprocessing techniques using multiple decoding units.

65. The method of claim 53 further comprising performing a hierarchical compressed capturing of the light field data, and performing a hierarchical decompression of the compressed light field data at the light field display system.

66. The method of claim 53 further comprising compressed capturing of a dynamic light field by utilizing temporal correlation.

67. The method of claim 66, further comprising motion estimation in the compressed capturing of the light field data.

68. The method of claim 67 wherein the compressed capturing uses depth or disparity image-based rendering, and further comprises reusing hardware and/or software that is used for the depth or disparity image-based rendering to perform both motion estimation and motion compensation.

69. The method of claim 53 further comprising time multiplexing the compressed light field data by rotations and/or translations of a light field emitter in the light field display system.

70. The method of claim 53 further comprising dividing the compressed light field data into hogel modulation groups that are used for parallel implementation of compressed rendering and display-matched encoding of the compressed capturing.

71. A method for light field imaging systems comprising:

compressed rendering of light field data from a light field to provide compressed rendered light field data; and

reconstructing and displaying the light field from the compressed rendered light field data at a light field display system.

72. A method for light field imaging systems comprising:

capturing light field data from a light field using display-matched encoding to provide display-matched compressed light field data; and

reconstructing and displaying the light field from the display-matched compressed light field data at a light field display system.