Video compression with adaptive view-dependent lighting removal

• US 10,567,464 B2
• Filed: 12/05/2017
• Issued: 02/18/2020
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Pang, Derek, Pitts, Colvin, Akeley, Kurt
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/832,023
• Publication Number:
  US 20180097867A1
• Time in Patent Office: 805 Days
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/349 Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for viewing without the aid of special glasses using fly-eye lenses H04N13/307
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/198 including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/44 : Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/46 : Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/553 : Motion estimation dealing with occlusions
• CPC Class Codes: H04N 19/597 : specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 21/816 : involving special video data, e.g 3D video
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 5/2226 : Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A video stream of a scene for a virtual reality or augmented reality experience may be captured by one or more image capture devices. Data from the video stream may be retrieved, including base vantage data with base vantage color data depicting the scene from a base vantage location, and target vantage data with target vantage color data depicting the scene from a target vantage location. The base vantage data may be reprojected to the target vantage location to obtain reprojected target vantage data. The reprojected target vantage data may be compared with the target vantage data to obtain residual data. The residual data may be compressed by removing a subset of the residual data that is likely to be less viewer-discernable than a remainder of the residual data. A compressed video stream may be stored, including the base vantage data and the compressed residual data.

Claims

1. A method for compressing a video stream for a virtual reality or augmented reality experience, the method comprising:

at one or more image capture devices, capturing a video stream of a scene;

at a data store, retrieving, from the video stream, base vantage data comprising base vantage color data depicting the scene from a base vantage location, and target vantage data comprising target vantage color data depicting the scene from a target vantage location;

at a processor, reprojecting the base vantage data to the target vantage location to obtain reprojected target vantage data;

at the processor, comparing the reprojected target vantage data with the target vantage data to obtain residual data;

at the processor, compressing the residual data to obtain compressed residual data by removing a subset of the residual data that is of a frequency that does not contribute to perceptual quality compared to a remainder of the residual data; and

in the data store, storing a compressed video stream comprising the base vantage data and the compressed residual data, wherein removing the subset of the residual data comprises applying entropy encoding to the residual data.

2. The method of claim 1, wherein removing the subset of the residual data comprises applying quantization to the residual data.

3. The method of claim 1, wherein removing the subset of the residual data comprises:

identifying an occluded region of the residual data, indicative of disocclusion between the base vantage location and the target vantage location; and

selecting the subset from outside the occluded region.

4. The method of claim 3, wherein removing the subset of the residual data further comprises generating a mask that delineates the occluded region and an outside region outside of the occluded region.

5. The method of claim 4, wherein removing the subset of the residual data further comprises removing all of the residual data in the outside region.

6. The method of claim 4, wherein removing the subset of the residual data further comprises:

applying a Gaussian smoothing kernel to blend the occluded region with the outside region to generate a blended occluded region; and

removing all of the residual data that lies outside the blended occluded region.

7. The method of claim 4, further comprising decoding the compressed video stream by using the mask to apply only the occluded region of the residual data to the base vantage data to reproject the base vantage data to the target vantage location.

8. The method of claim 1, wherein removing the subset of the residual data comprises removing reprojection errors from the residual data.

9. The method of claim 1, wherein removing the subset of the residual data comprises removing view-dependent lighting from the residual data.

10. The method of claim 9, wherein removing the subset of the residual data further comprises removing all of the view-dependent lighting from the residual data.

11. The method of claim 9, wherein removing the subset of the residual data further comprises:

identifying a subset of the view-dependent lighting that is to be of the frequency that does not contribute to perceptual quality; and

removing only the subset from the residual data.

12. The method of claim 1, further comprising decoding the compressed video stream by applying the remainder of the residual data to the base vantage data to reproject the base vantage data to the target vantage location.

13. A non-transitory computer-readable medium for compressing a video stream for a virtual reality or augmented reality experience, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing a data store to retrieve, from a video stream of a scene captured by one or more image capture devices, base vantage data comprising base vantage color data depicting the scene from a base vantage location, and target vantage data comprising target vantage color data depicting the scene from a target vantage location;

reprojecting the base vantage data to the target vantage location to obtain reprojected target vantage data;

comparing the reprojected target vantage data with the target vantage data to obtain residual data;

compressing the residual data to obtain compressed residual data by removing a subset of the residual data that is of a frequency that does not contribute to perceptual quality compared to a remainder of the residual data; and

causing the data store to store a compressed video stream comprising the base vantage data and the compressed residual data, wherein removing the subset of the residual data comprises applying entropy encoding to the residual data.

14. The non-transitory computer-readable medium of claim 13, wherein removing the subset of the residual data comprises:

identifying an occluded region of the residual data, indicative of disocclusion between the base vantage location and the target vantage location;

selecting the subset from outside the occluded region; and

generating a mask that delineates the occluded region and an outside region outside of the occluded region.

15. The non-transitory computer-readable medium of claim 14, wherein removing the subset of the residual data further comprises removing all of the residual data in the outside region.

16. The non-transitory computer-readable medium of claim 14, wherein removing the subset of the residual data further comprises:

applying a Gaussian smoothing kernel to blend the occluded region with the outside region to generate a blended occluded region; and

removing all of the residual data that lies outside the blended occluded region.

17. The non-transitory computer-readable medium of claim 14, further comprising instructions stored thereon, that when executed by the processor, decodes the compressed video stream by using the mask to apply only the occluded region of the residual data to the base vantage data to reproject the base vantage data to the target vantage location.

18. The non-transitory computer-readable medium of claim 13, wherein removing the subset of the residual data comprises removing reprojection errors from the residual data.

19. The non-transitory computer-readable medium of claim 13, wherein removing the subset of the residual data comprises removing all view-dependent lighting from the residual data.

20. The non-transitory computer-readable medium of claim 13, wherein removing the subset of the residual data comprises:

identifying a subset of view-dependent lighting of the residual data that is to be of the frequency that does not contribute to perceptual quality; and

removing only the subset from the residual data.

21. The non-transitory computer-readable medium of claim 13, further comprising instructions stored thereon, that when executed by the processor, decodes the compressed video stream by applying the remainder of the residual data to the base vantage data to reproject the base vantage data to the target vantage location.

22. A system for compressing a video stream for a virtual reality or augmented reality experience, the system comprising:

one or more image capture devices configured to capture a video stream of a scene;

a data store configured to retrieve, from the video stream, base vantage data comprising base vantage color data depicting the scene from a base vantage location, and target vantage data comprising target vantage color data depicting the scene from a target vantage location; and

a processor configured to;

reproject the base vantage data to the target vantage location to obtain reprojected target vantage data;

compare the reprojected target vantage data with the target vantage data to obtain residual data; and

compress the residual data to obtain compressed residual data by removing a subset of the residual data that is of a frequency that does not contribute to perceptual quality compared to a remainder of the residual data;

wherein the data store is further configured to store a compressed video stream comprising the base vantage data and the compressed residual data, wherein removing the subset of the residual data comprises applying entropy encoding to the residual data.

23. The system of claim 22, wherein the processor is further configured to remove the subset of the residual data by:

identifying an occluded region of the residual data, indicative of disocclusion between the base vantage location and the target vantage location;

selecting the subset from outside the occluded region; and

generating a mask that delineates the occluded region and an outside region outside of the occluded region.

24. The system of claim 23, wherein the processor is further configured to remove the subset of the residual data by removing all of the residual data in the outside region.

25. The system of claim 23, wherein the processor is further configured to remove the subset of the residual data by:

applying a Gaussian smoothing kernel to blend the occluded region with the outside region to generate a blended occluded region; and

removing all of the residual data that lies outside the blended occluded region.

26. The system of claim 23, wherein the processor is further configured to decode the compressed video stream by using the mask to apply only the occluded region of the residual data to the base vantage data to reproject the base vantage data to the target vantage location.

27. The system of claim 22, wherein the processor is further configured to remove the subset of the residual data by removing reprojection errors from the residual data.

28. The system of claim 22, wherein removing the subset of the residual data comprises removing all view-dependent lighting from the residual data.

29. The system of claim 22, wherein removing the subset of the residual data further comprises:

identifying a subset of view-dependent lighting of the residual data that is to be of the frequency that does not contribute to perceptual quality; and

removing only the subset from the residual data.

30. The system of claim 22, wherein the processor is further configured to decode the compressed video stream by applying the remainder of the residual data to the base vantage data to reproject the base vantage data to the target vantage location.