MOTION-COMPENSATED COMPRESSION OF DYNAMIC VOXELIZED POINT CLOUDS

• US 20170347120A1
• Filed: 05/28/2016
• Published: 11/30/2017
• Est. Priority Date: 05/28/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chou, Philip A., de Queiroz, Ricardo L.
• Granted Patent: US 11,297,346 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06T 9/004 Predictors, e.g. intraframe, interframe coding
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/119 Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/19 using optimisation based on Lagrange multipliers
• CPC Class Codes: H04N 19/43 Hardware specially adapted for motion estimation or compensation
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder

Abstract

Disclosed herein are exemplary embodiments of innovations in the area of point cloud encoding and decoding. Example embodiments can reduce the computational complexity and/or computational resource usage during 3D video encoding by selectively encoding one or more 3D-point-cloud blocks using an inter-frame coding (e.g., motion compensation) technique that allows for previously encoded/decoded frames to be used in predicting current frames being encoded. Alternatively, one or more 3D-point-cloud block can be encoded using an intra-frame encoding approach. The selection of which encoding mode to use can be based, for example, on a threshold that is evaluated relative to rate-distortion performance for both intra-frame and inter-frame encoding. Still further, embodiments of the disclosed technology can use one or more voxel-distortion-correction filters to correct distortion errors that may occur during voxel compression. Such filters are uniquely adapted for the particular challenges presented when compressing 3D image data. Corresponding decoding techniques are also disclosed.

Claims

1. A video encoder system, comprising:

a buffer configured to store voxelized point cloud data of a sequence of three-dimensional video frames to be encoded; and

a video encoder configured to encode the frames of the sequence by;

selecting an encoding mode for one or more occupied 3D-point-cloud blocks in a current frame of the sequence, the encoding mode being one of a plurality of available encoding modes, the available encoding modes including intra-frame encoding and inter-frame encoding;

encoding the one or more occupied 3D-point-cloud blocks of the current frame of the sequence according to the selected encoding mode; and

outputting a bitstream comprising encoded data for the one or more occupied 3D-point-cloud blocks.

2. The video encoder system of claim 1, wherein the bitstream includes a syntax element or bitstream element signaling the selected encoding mode for respective 3D-point-cloud blocks.

3. The video encoder system of claim 1, wherein the inter-frame encoding mode uses motion compensation.

4. The video encoder system of claim 3, wherein a motion prediction used for the motion compensation is block translational, affine, or non-rigid.

5. The video encoder system of claim 3, wherein the motion compensation uses block replenishment.

6. The video encoder system of claim 3, wherein the block replenishment comprises:

matching an occupied 3D-point-cloud block from the current frame with a corresponding occupied 3D-point-cloud block from a previous frame using motion vector data; and

encoding the motion vector data for the occupied 3D-point-cloud block.

7. The video encoder system of claim 3, wherein the block replenishment excludes encoding of geometry residuals.

8. The video encoder system of claim 1, wherein the selecting of the encoding mode is based at least in part on a correspondence-based metric, a projection-based metric, or a combination of both the correspondence-based metric and the projection-based metric.

9. The video encoder system of claim 1, wherein the selecting of the encoding mode is based at least in part on a metric that includes both geometry and color components.

10. The video encoder system of claim 1, wherein the selecting is based on a threshold that relates rate-distortion performance for intra-frame encoding to rate-distortion performance for inter-frame encoding.

11. One or more computer-readable memory or storage devices storing computer-executable instructions which when executed by a computing device causes the computing device to perform encoding operations comprising:

encoding three-dimensional image data for a current frame in a sequence of three-dimensional video frames, the frame being formed from a plurality of 3D-point-cloud blocks, the encoding including encoding occupied ones of the 3D-point-cloud blocks; and

outputting a bitstream comprising the encoded three-dimensional image data.

12. The one or more computer-readable memory or storage devices of claim 11, wherein each 3D-point-cloud block includes a plurality of voxels.

13. The one or more computer-readable memory or storage devices of claim 11, wherein the encoding further comprises applying one or more voxel-distortion-correction filters to the three-dimensional image data for the current frame, the one or more voxel-distortion-correction filters being applied in-loop during inter-frame encoding.

14. The one or more computer-readable memory or storage devices of claim 13, wherein the one or more voxel-distortion-correction filters comprise a morphological process.

15. The one or more computer-readable memory or storage devices of claim 14, wherein the morphological process comprises:

a dilation phase; and

an erosion phase.

16. The one or more computer-readable memory or storage devices of claim 15, wherein the dilation phase comprises replicating one or more existing occupied voxels volumetrically to volumetric neighbors.

17. The one or more computer-readable memory or storage devices of claim 13, wherein the one or more voxel-distortion-correction filters include an adaptive smoothing process.

18. A method comprising:

by a computing device implementing a decoder for three-dimensional video data;

receiving a bitstream of encoded dynamic point cloud data for a sequence of frames, the bitstream further comprising one or more syntax values or bitstream elements signaling an encoding mode for one or more 3D-point-cloud blocks in a current frame in the sequence; and

decoding the encoded data for the sequence of frames, including decoding the one or more of the 3D-point-cloud blocks of the current frame in accordance with the signaled encoding mode.

19. The method of claim 18, wherein the signaled encoding mode is one of multiple available encoding modes, the multiple available encoding modes includes at least an intra-frame encoding mode and an inter-frame encoding mode.

20. The method of claim 18, wherein the signaled encoding mode indicates inter-frame encoding for the one or more 3D-point-cloud blocks, and wherein the decoding comprises:

decoding motion vector data for the one or more 3D-point-cloud blocks, the motion vector data indicating three-dimensional translational movement of voxels from a previous frame; and

decoding the one or more 3D-point-cloud block using the voxels from the previous frame translated according to the motion vector data as predictors of voxels in the one or more 3D-point-cloud blocks being decoded.