Point cloud compression with closed-loop color conversion

• US 10,607,373 B2
• Filed: 11/21/2018
• Issued: 03/31/2020
• Est. Priority Date: 11/22/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mammou, Khaled, Su, Yeping, Singer, David W., Tourapis, Alexandros, Robinet, Fabrice A., Wu, Hsi-Jung
• Primary Examiner(s): Broome, Said
• Application Number:
  US16/198,630
• Publication Number:
  US 20190156518A1
• Time in Patent Office: 496 Days
• Field of Search: None
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00

Abstract

A system comprises an encoder configured to compress attribute information and/or spatial for a point cloud and/or a decoder configured to decompress compressed attribute and/or spatial information for the point cloud. To compress the attribute and/or spatial information, the encoder is configured to convert a point cloud into an image based representation. Also, the decoder is configured to generate a decompressed point cloud based on an image based representation of a point cloud. A closed-loop color conversion process is used to improve compression while taking into consideration distortion introduced throughout the point cloud compression process.

Claims

1. A system comprising:

an encoder configured to compress a point cloud, wherein to compress the point cloud, the encoder is configured to;

determine for the point cloud a plurality of patches, and for each patch,

generate one or more patch images comprising a set of points corresponding to the patch projected onto a patch plane, wherein the one or more patch images comprise color information for the set of points; and

convert at least one of the patch images from a first color space to a second color space;

pack generated patch images for each of the determined patches into one or more image frames; and

encode the one or more image frames,

wherein the encoder is configured to select one or more parameters used to convert the patch image from the first color space to the second color space to compensate for distortion introduced due to compression of the point cloud.

2. The system of claim 1, wherein the distortion introduced due to compression of the point cloud comprises distortion caused by:

generating the one or more patch images and packing the generated one or more patch images in the one or more image frames;

encoding the one or more image frames; and

converting the patch image from the first color space to the second color space.

3. The system of claim 1, wherein the one or more parameters are selected using a closed-form solution approach that minimizes distortion of a reconstructed point cloud having point colors assigned based on the at least one color converted patch image.

4. The system of claim 1, wherein the first color space is a red, green, blue (RGB) color space, and wherein the second color space is a Luma, blue difference chroma, and red-difference chroma (YCbCr) color space.

5. The system of claim 4, wherein the encoder is further configured to convert the at least one of the patch images from an RGB 4:

4;

4 representation to a YCbCr 4;

2;

0 representation.

6. The system of claim 5, wherein the encoder is configured to select a luma value (Y) that minimizes the distortion introduced due to compression of the point cloud.

7. The system of claim 1, wherein the one or more parameters are selected using an iterative solution approach that minimizes distortion of a reconstructed point cloud having point colors assigned based on the color converted patch images.

8. The system of claim 1, wherein to select the one or more parameters used to convert the patch image from the first color space to the second color space, the encoder is configured to:

match points in the point cloud prior to compression to corresponding points in a reconstructed version of the point cloud;

determine geometry distortion or texture distortion of the reconstructed version of the point cloud in comparison to the point cloud prior to compression; and

select the one or more parameters based at least in part on the determined geometry distortion or texture distortion.

9. A method for converting a patch image of a point cloud being compressed by an encoder from a first color space to a second color space, the method comprising:

reconstructing, by a reconstruction module of the encoder, at least a portion of the point cloud using the patch image and one or more additional patch images;

determining geometry distortion or texture distortion of the reconstructed point cloud in comparison to the point cloud prior to compression;

selecting a color conversion parameter for converting the patch image being compressed by the point cloud encoder from the first color space to the second color space, wherein the selected color conversion parameter is selected based on the determined geometry distortion or texture distortion, and wherein the color conversion parameter is selected to reduce distortion of a reconstructed point cloud to be reconstructed by a decoder using the compressed patch image.

10. The method of claim 9, further comprising:

determining color distortion for an image frame comprising the patch image being compressed by the point cloud encoder, wherein the color conversion parameter is selected, based at least in part, on the determined color distortion for the image frame.

11. The method of claim 10, wherein color distortion of the image frame is determined on a point-wise basis, and wherein the color conversion parameter is selected to reduce overall color distortion of points of the patch image.

12. The method of claim 10, wherein color distortion of the image frame is determined on a point-neighborhood basis, wherein different weighting values are associated with different point neighborhoods, and wherein the color conversion parameter is selected to reduce a weighted color distortion parameter for the different point neighborhoods.

13. The method of claim 12, further comprising:

determining areas of the image frame that correspond to the patch image and areas of the image frame that correspond to padding,

wherein color distortion of the portions of the image frame corresponding to padding are given a lesser weight in selecting the color conversion parameter than color distortion of portions of the image frame corresponding to the patch image.

14. The method of claim 9, further comprising:

repeating said reconstructing the point cloud and said determining the geometry distortion or texture distortion of the reconstructed point cloud using one or more additional decoders of a different decoder type than the decoder; and

assigning respective weighting values to geometry distortion or texture distortion introduced by the decoder and the one or more additional decoders,

wherein the color conversion parameter is selected based, at least in part, on a weighted distortion value for the decoder and the one or more additional decoders.

15. The method of claim 9, further comprising:

determining a type of decoder likely to be used to reconstruct a compressed point cloud comprising the compressed patch image,

wherein the determined type of decoder is used to reconstruct the point cloud to determine the geometry distortion or the texture distortion.

16. The method of claim 9, further comprising:

determining for the point cloud a plurality of patches; and

generating the patch image, wherein the patch image comprises a set of points of the point cloud corresponding to one of the patches projected onto a patch plane.

17. The method of claim 16, further comprising:

compressing the image frame using a lossy video endcoder, wherein the selection of the color conversion parameter takes into account distortion caused by the lossy video encoder.

18. A non-transitory computer readable medium storing program instruction that, when executed by one or more processors, cause the one or more processors to:

reconstruct a point cloud using a patch image and one or more additional patch images;

determine geometry distortion or texture distortion of the reconstructed point cloud in comparison to the point cloud prior to compression; and

select a color conversion parameter for converting the patch image from a first color space to a second color space, wherein the selected color conversion parameter is selected based on the determined geometry distortion or texture distortion, and wherein the color conversion parameter is selected to reduce distortion of a reconstructed point cloud to be reconstructed by a decoder using the compressed patch image.

19. The non-transitory computer readable medium of claim 18, wherein the first color space is an RGB color space and the second color space is a Luma, blue difference chroma, and red-difference chroma (YCbCr) color space.

20. The non-transitory computer readable medium of claim 19, wherein the selected color conversion parameter is a luma value (Y) that minimizes distortion.