Volumetric video compression with motion history

• US 10,708,627 B2
• Filed: 03/04/2019
• Issued: 07/07/2020
• Est. Priority Date: 03/04/2019
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Lucas, Blake C., Lavrov, Dmitry
• Primary Examiner(s): Nawaz, Talha M
• Application Number:
  US16/292,176
• Publication Number:
  US 20190200046A1
• Time in Patent Office: 491 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06T 9/004 Predictors, e.g. intraframe, interframe coding
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 19/10 using adaptive coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Abstract

Techniques related to pose estimation for an articulated body are discussed. Such techniques may include extracting, segmenting, classifying, and labeling blobs, generating initial kinematic parameters that provide spatial relationships of elements of a kinematic model representing an articulated body, and refining the kinematic parameters to provide a pose estimation for the articulated body.

Claims

1. A system for compressing 3D video comprising:

memory to store first and second point clouds for a current and a previous time instance, respectively; and

a processor coupled to the memory, the processor to;

receive a delta frame representative of the first 3D point cloud, wherein the delta frame references a key frame representative of the second 3D point cloud, wherein each point of the first 3D point cloud has a delta frame index value and each point of the second 3D point cloud has a key frame index value;

receive an index point correlation mapping between the first and second 3D point clouds, wherein the index point correlation mapping comprises at least a plurality of index point correlations, each of the plurality of index point correlations matching one of the delta frame index values with a matching one of the key frame index values;

generate an index point correlation differences list by differencing each of the index point correlations;

run length encode the index point correlation differences list to generate a run length encoded index point correlation differences list; and

entropy encode the run length encoded index point correlation differences list and the delta frame into a bitstream.

2. The system of claim 1, wherein the index point correlation mapping provides an injective mapping between the first and second 3D point clouds, the plurality of index point correlations comprises a first subset of index point correlations having equivalent delta frame and key frame index values run length encoded as a run of zeros, a second subset of index point correlations having different delta frame and key frame index values run length encoded as the difference between the delta frame and key frame index values, and a plurality of delta frame index values each matched to a particular value indicative of no matching point in the second point cloud.

3. The system of claim 1, wherein each point of the first 3D point cloud corresponds to a quantized and coded point of a first raw 3D point cloud and each point of the second 3D point cloud corresponds to a quantized and coded point of a second raw 3D point cloud.

4. The system of claim 1, wherein the delta frame comprises a spatial displacement for each of the matching ones of the key frame index values, each spatial displacement comprising 3D displacement between the point of the second 3D point cloud for the matching one of the key frame index value and the point of the first 3D point cloud for the matching one of the delta frame index values.

5. The system of claim 1, wherein the delta frame comprises a color residual for at least one of the matching ones of the key frame index values, the color residual comprising color channel residual values between color channel values corresponding to the point of the second 3D point cloud for the matching one of the key frame index value and the point of the first 3D point cloud for the matching one of the delta frame index values.

6. The system of claim 1, further comprising the processor to generate the key frame by color compression of a 3D block corresponding to the key frame, the color compression comprising the processor to:

determine color values for each of a plurality of points within the 3D block;

project the color values onto a first face and a second face of the 3D block, wherein said projection of the color values onto the first face comprises the processor to combine color values for two of the plurality of points that are aligned with a direction orthogonal to the first face;

project the color values from the first face back to the plurality of points and determine a first error corresponding thereto;

project the color values from the second face back to the plurality of points and determine a second error corresponding thereto; and

encode, in response to the first error being less than the second error and less than a threshold, the projected color values of the first face for the 3D block.

7. The system of claim 6, wherein the processor to encode the projected color values of the first face for the 3D block comprises the processor to transform the projected color values of the first face to the frequency domain.

8. The system of claim 6, wherein, when the first error and the second error exceed the threshold, the color values for each of the plurality of points are encoded for the 3D block.

9. The system of claim 1, wherein further comprising the processor to generate the delta frame by color compression a 3D block corresponding to the delta frame, the color compression comprising the processor to:

determine residual color values for each of a plurality of points within the 3D block;

project the residual color values onto a first face and a second face of the 3D block by eliminating one spatial dimension of each of the plurality of points for each projection and combining any residual color values for points aligned along the eliminated spatial dimension;

project the color values from the first and second faces back to the plurality of points and determining first and second errors corresponding thereto; and

encode, in response to the first error being less than the second error and less than a threshold, the projected residual color values of the first face for the 3D block, or

encode, in response to the first and second errors both exceeding the threshold, the residual color values for the 3D block.

10. The system of claim 9, wherein the processor to encode the projected residual color values of the first face for the 3D block comprises the processor to apply a discrete cosine transform to the projected residual color values.

11. A method for compressing 3D video comprising:

receiving a delta frame representative of a first 3D point cloud for a current time instance, wherein the delta frame references a key frame representative of a second 3D point cloud for a previous time instance, wherein each point of the first 3D point cloud has a delta frame index value and each point of the second 3D point cloud has a key frame index value;

receiving an index point correlation mapping between the first and second 3D point clouds, wherein the index point correlation mapping comprises at least a plurality of index point correlations, each of the plurality of index point correlations matching one of the delta frame index values with a matching one of the key frame index values;

generating an index point correlation differences list by differencing each of the index point correlations;

run length encoding the index point correlation differences list to generate a run length encoded index point correlation differences list; and

entropy encoding the run length encoded index point correlation differences list and the delta frame into a bitstream.

12. The method of claim 11, wherein the index point correlation mapping provides an injective mapping between the first and second 3D point clouds, the plurality of index point correlations comprises a first subset of index point correlations having equivalent delta frame and key frame index values run length encoded as a run of zeros, a second subset of index point correlations having different delta frame and key frame index values run length encoded as the difference between the delta frame and key frame index values, and a plurality of delta frame index values each matched to a particular value indicative of no matching point in the second point cloud.

13. The method of claim 11, wherein the delta frame comprises a spatial displacement for each of the matching ones of the key frame index values, each spatial displacement comprising 3D displacement between the point of the second 3D point cloud for the matching one of the key frame index value and the point of the first 3D point cloud for the matching one of the delta frame index values.

14. The method of claim 11, wherein the delta frame comprises a color residual for at least one of the matching ones of the key frame index values, the color residual comprising color channel residual values between color channel values corresponding to the point of the second 3D point cloud for the matching one of the key frame index value and the point of the first 3D point cloud for the matching one of the delta frame index values.

15. The method of claim 11, further comprising generating the key frame by color compressing a 3D block corresponding to the key frame by:

determining color values for each of a plurality of points within the 3D block;

projecting the color values onto a first face and a second face of the 3D block, wherein said projecting the color values onto the first face comprises combining color values for two of the plurality of points that are aligned with a direction orthogonal to the first face;

projecting the color values from the first face back to the plurality of points and determining a first error corresponding thereto;

projecting the color values from the second face back to the plurality of points and determining a second error corresponding thereto; and

encoding, in response to the first error being less than the second error and less than a threshold, the projected color values of the first face for the 3D block.

16. The method of claim 11 further comprising generating the delta frame by color compressing a 3D block corresponding to the delta frame by:

determining residual color values for each of a plurality of points within the 3D block;

projecting the residual color values onto a first face and a second face of the 3D block by eliminating one spatial dimension of each of the plurality of points for each projection and combining any residual color values for points aligned along the eliminated spatial dimension;

projecting the color values from the first and second faces back to the plurality of points and determining first and second errors corresponding thereto; and

encoding, in response to the first error being less than the second error and less than a threshold, the projected residual color values of the first face for the 3D block, or

encoding, in response to the first and second errors both exceeding the threshold, the residual color values for the 3D block.

17. At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to compress 3D video by:

receiving a delta frame representative of a first 3D point cloud for a current time instance, wherein the delta frame references a key frame representative of a second 3D point cloud for a previous time instance, wherein each point of the first 3D point cloud has a delta frame index value and each point of the second 3D point cloud has a key frame index value;

receiving an index point correlation mapping between the first and second 3D point clouds, wherein the index point correlation mapping comprises at least a plurality of index point correlations, each of the plurality of index point correlations matching one of the delta frame index values with a matching one of the key frame index values;

generating an index point correlation differences list by differencing each of the index point correlations;

run length encoding the index point correlation differences list to generate a run length encoded index point correlation differences list; and

entropy encoding the run length encoded index point correlation differences list and the delta frame into a bitstream.

18. The non-transitory machine readable medium of claim 17, wherein the index point correlation mapping provides an injective mapping between the first and second 3D point clouds, the plurality of index point correlations comprises a first subset of index point correlations having equivalent delta frame and key frame index values run length encoded as a run of zeros, a second subset of index point correlations having different delta frame and key frame index values run length encoded as the difference between the delta frame and key frame index values, and a plurality of delta frame index values each matched to a particular value indicative of no matching point in the second point cloud.

19. The non-transitory machine readable medium of claim 17, wherein the delta frame comprises a spatial displacement for each of the matching ones of the key frame index values, each spatial displacement comprising 3D displacement between the point of the second 3D point cloud for the matching one of the key frame index value and the point of the first 3D point cloud for the matching one of the delta frame index values.

20. The non-transitory machine readable medium of claim 17, wherein the delta frame comprises a color residual for at least one of the matching ones of the key frame index values, the color residual comprising color channel residual values between color channel values corresponding to the point of the second 3D point cloud for the matching one of the key frame index value and the point of the first 3D point cloud for the matching one of the delta frame index values.

21. The non-transitory machine readable medium of claim 17, comprising further instructions that, in response to being executed on the computing device, cause the computing device to generate the key frame by color compressing a 3D block corresponding to the key frame by:

determining color values for each of a plurality of points within the 3D block;

projecting the color values onto a first face and a second face of the 3D block, wherein said projecting the color values onto the first face comprises combining color values for two of the plurality of points that are aligned with a direction orthogonal to the first face;

projecting the color values from the first face back to the plurality of points and determining a first error corresponding thereto;

projecting the color values from the second face back to the plurality of points and determining a second error corresponding thereto; and

encoding, in response to the first error being less than the second error and less than a threshold, the projected color values of the first face for the 3D block.

22. The non-transitory machine readable medium of claim 17, comprising further instructions that, in response to being executed on the computing device, cause the computing device to generate the delta frame by color compressing a 3D block corresponding to the key frame by:

determining residual color values for each of a plurality of points within the 3D block;

projecting the residual color values onto a first face and a second face of the 3D block by eliminating one spatial dimension of each of the plurality of points for each projection and combining any residual color values for points aligned along the eliminated spatial dimension;

projecting the color values from the first and second faces back to the plurality of points and determining first and second errors corresponding thereto; and

encoding, in response to the first error being less than the second error and less than a threshold, the projected residual color values of the first face for the 3D block, or

encoding, in response to the first and second errors both exceeding the threshold, the residual color values for the 3D block.