POINT CLOUD COMPRESSION

• US 20190311500A1
• Filed: 04/10/2019
• Published: 10/10/2019
• Est. Priority Date: 04/10/2018
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mammou, Khaled, Valentin, Valery G., Robinet, Fabrice A., Tourapis, Alexandros, Su, Yeping, Kim, Jungsun
• Granted Patent: US 10,909,726 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame

Abstract

A system comprises an encoder configured to compress attribute information and/or spatial for a point cloud and/or a decoder configured to decompress compressed attribute and/or spatial information for the point cloud. To compress the attribute and/or spatial information, the encoder is configured to convert a point cloud into an image based representation. Also, the decoder is configured to generate a decompressed point cloud based on an image based representation of a point cloud. In some embodiments, an encoder may be configured to further compress points omitted from the image based representation. Also, in some embodiments, a decoder may be configured to decode points compressed outside of an image based representation or in a separate image based representation.

Claims

1. A system comprising:

one or more sensors configured to capture a plurality of points that make up a point cloud, wherein respective ones of the points comprise spatial information for the point and attribute information for the point; and

an encoder configured to compress the point cloud, wherein to compress the point cloud, the encoder is configured to;

determine a plurality of patches each corresponding to portions of the point cloud;

for each patch, generate a patch image comprising attribute information or spatial information for a set of points corresponding to a patch projected onto a patch plane;

pack generated patch images for each of the determined patches into one or more patch image frames;

generate a reconstructed version of the point cloud based on the patch images;

determine one or more points included in the point cloud that are omitted from the reconstructed version of the point cloud; and

encode the one or more patch image frames and spatial or attribute information for the one or more omitted points, wherein the one or more omitted points are compressed without determining additional patches for the one or more omitted points.

2. The system of claim 1, wherein to encode the spatial information for the one or more omitted points, the encoder is configured to, for each of the one or more omitted points:

identify a point in the reconstructed version of the point cloud neighboring the omitted point; and

encode spatial information indicating a position of the omitted point relative to the neighboring point.

3. The system of claim 2, wherein information for the the one or more identified points neighboring the one or more omitted points is encoded using:

a prediction strategy to identify the one or more neighboring points from points in the reconstructed point cloud, reconstructed from the patches packed into the one or more image frames, wherein information for use in the prediction strategy to identify the one or more neighboring points from the points in the reconstructed point cloud is encoded according to an arithmetic encoding technique;

or

a quadtree encoding technique to identify the one or more neighboring point from points in the reconstructed point cloud, wherein a binary code generated for the quadtree is entropy encoded.

4. The system of claim 3, wherein to encode the attribute information for the one or more omitted points, the encoder is configured to, for each of the one or more omitted points:

identify a point in the reconstructed version of the point cloud neighboring the omitted point; and

encode attribute information indicating residual differences between one or more attribute values of the neighboring point and attribute values of the omitted point.

5. The system of claim 4, wherein to encode the attribute information indicating residual differences or the spatial information indicating a position of the omitted point relative to the neighboring point, the encoder is configured to:

predict attribute residual differences or spatial residual differences for the omitted point relative to the neighboring point based on attribute values or spatial coordinates of the neighboring point and a set of other neighboring points in the reconstructed point cloud or previously encoded and decoded omitted points;

determine correction values that account for differences between the predicted attribute residual differences or spatial residual differences and actual attribute residual differences or actual spatial residual differences between the omitted point and the neighboring point; and

entropy encode the determined correction values.

6. The system of claim 4, wherein to encode the spatial information or the attribute information for the one or more omitted points, the encoder is configured to:

encode the spatial information or the attribute information for the one or more omitted points in another image frame, wherein the one or more patch image frames and the other image frame are encoded according to a high efficiency video codec.

7. The system of claim 6, wherein the encoder is configured to select a frame size for the other image frame comprising the one or more omitted points based on a number of omitted points to be encoded, wherein depending on the number of omitted points to be encoded the frame size for the other image frame varies from a frame size of the one or more patch image frames.

8. A method comprising:

determining, for a point cloud, a plurality of patches each corresponding to portions of the point cloud;

for each patch, generating a patch image comprising attribute information or spatial information for a set of points corresponding to a patch projected onto a patch plane;

packing generated patch images for each of the determined patches into one or more patch image frames;

generating a reconstructed version of the point cloud based on the patch images;

determining one or more points included in the point cloud that are omitted from the reconstructed version of the point cloud; and

encoding the one or more patch image frames and spatial or attribute information for the one or more omitted points, wherein the one or more omitted points are compressed without determining additional patches for the one or more omitted points.

9. The method of claim 8, wherein encoding the spatial information or the attribute information for the one or more omitted points comprises:

encoding the spatial information or the attribute information for the one or more omitted points in another image frame, wherein the one or more patch image frames and the other image frame are encoded according to a high efficiency video codec.

10. The method of claim 9, wherein absolute values for spatial coordinates for the one or more omitted points or absolute values for attributes of the one or more omitted points are encoded in the other image frame.

11. The method of claim 9, wherein encoding the spatial information or the attribute information for the one or more omitted points comprises, for each of the one or more omitted points:

identifying a point in the reconstructed version of the point cloud neighboring the omitted point; and

encoding attribute information or spatial information indicating residual differences between one or more attribute values of the neighboring point and attribute values of the omitted point or indicating residual differences between spatial coordinates of the neighboring point and the omitted point,

wherein the spatial information or the attribute information for the one or more omitted points encoded in the other image frame comprises spatial residual differences or attribute residual differences between the omitted point and the neighboring point.

12. The method of claim 11, wherein:

a first one of the one or more omitted points has different coordinates values than the neighboring point in a tangential or bi-tangential direction on the patch projection plane, wherein for the first one of the one or more omitted points, residual differences for two or more spatial coordinates are encoded; and

a second one of the one or more omitted points has a different depth value than the neighboring point, but a same position in the projection plane, wherein for the second omitted point a residual depth value is encoded without encoding other residual differences for other spatial coordinates.

13. The method of claim 12, wherein:

depth only omitted points are encoded in a particular sequence of pixels of the other image frame, wherein one pixel per omitted point is encoded to indicate the residual depth values of the respective depth only omitted points; and

multi-coordinate omitted points are encoded in another sequence of pixels of the other image frame, wherein two or more pixels are encoded per multi-coordinate omitted point to indicate residual coordinate values in two or more coordinate directions.

14. The method of claim 13, further comprising:

selecting a scanning order for encoding pixels of the other image frame according to a rate distortion optimization analysis; and

signaling the selected scanning order along with the encoded spatial or attribute information for the one or more omitted points.

15. A non-transitory computer-readable medium storing program instructions, that when executed by one or more processors, cause the one or more processors to:

determine, for a point cloud, a plurality of patches each corresponding to portions of the point cloud;

for each patch, generate a patch image comprising attribute information or spatial information for a set of points corresponding to a patch projected onto a patch plane;

pack generated patch images for each of the determined patches into one or more patch image frames;

generate a reconstructed version of the point cloud based on the patch images;

determine one or more points included in the point cloud that are omitted from the reconstructed version of the point cloud; and

encode the one or more patch image frames and spatial or attribute information for the one or more omitted points, wherein the one or more omitted points are compressed without determining additional patches for the one or more omitted points.

16. The non-transitory computer-readable medium of claim 15, wherein to encode the spatial information for the one or more omitted points, the program instructions, when executed on the one or more processors, cause the one or more processors to:

identify, for each of the one or more omitted points, a point in the reconstructed version of the point cloud neighboring the omitted point; and

encode information indicating a respective neighboring point for each of the one or more omitted points of a given patch image frame according to a quadtree encoding technique, wherein a binary code generated for the quadtree is entropy encoded.

17. The non-transitory computer-readable medium of claim 15, wherein to encode the attribute information or the spatial information for the one or more omitted points, the program instructions, when executed on the one or more processors, cause the one or more processors to:

encode the spatial information or the attribute information for the one or more omitted points in another image frame, wherein the one or more patch image frames and the other image frame are encoded according to a high efficiency video codec.

18. The non-transitory computer-readable medium of claim 17, wherein the spatial information or the attribute information for the one or more omitted points is organized in a plurality of blocks of the other image frame, wherein the program instructions cause the one or more processors to:

signal a block size or a number of blocks for blocks of the other image frame.

19. The non-transitory computer-readable medium of claim 17, wherein the program instructions cause the one or more processors to:

signal a scanning order for the pixels of the other image frame.

20. The non-transitory computer-readable medium of claim 17, wherein the program instructions cause the one or more processors to:

signal whether residual coordinate spatial values are organized in a scanning order based on common coordinates or are organized in the scanning order sequentially for each omitted point,

wherein the encoder selects the scanning order selected based on a rate distortion optimization analysis.