SCALABLE COMPRESSION OF TIME-CONSISTEND 3D MESH SEQUENCES

• US 20120262444A1
• Filed: 04/18/2008
• Published: 10/18/2012
• Est. Priority Date: 04/18/2007
• Status: Active Grant

• Current Assignee: Gottfried Wilhelm Leibniz Universitã¤T Hannover
• Original Assignee: Gottfried Wilhelm Leibniz Universitã¤T Hannover
• Inventors: Stefanoski, Nikolce, Ostermann, Jörn, Klie, Patrick
• Granted Patent: US 8,502,815 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame

Abstract

We present a method for predictive compression of time-consistent 3D mesh sequences supporting and exploiting scalability. The applied method decomposes each frame of a mesh sequence in layers, which provides a time-consistent multi-resolution representation. Following the predictive coding paradigm, local temporal and spatial dependencies between layers and frames are exploited for layer-wise compression. Prediction is performed vertex-wise from coarse to fine layers exploiting the motion of already encoded neighboring vertices for prediction of the current vertex location. Consequently, successive layer-wise decoding allows to reconstruct frames with increasing levels of detail.

Claims

1-63. -63. (canceled)

64. Method for encoding a time-consistent 3D mesh sequence using scalable predictive coding, the method comprising the steps of:

receiving successive frames of a 3D mesh sequence, wherein a frame comprises a set of vertices (V) with associated connectivity (K) and wherein the connectivity (K) is identical for all frames of the 3D mesh sequence,

decomposing the 3D mesh sequence into a multi-resolution representation comprising two or more spatial layers represented by disjoint sets of vertices (V_l) with associated connectivities (K_l) and

encoding the vertices of the layers using a predictive encoder,

wherein the decomposition is calculated from the connectivity (K) of the 3D mesh sequence irrespectively of location data of the vertices.

65. The method according to claim 64, further comprising the steps:

encoding the connectivity (K) of the 3D mesh sequence for a first frame of the sequence, and

selecting for each frame a compression method from a group of compression methods comprising I-frame compression and P-frame compression and/or B-frame compression.

66. The method according to claim 64 or 65, wherein the step of decomposing the 3D mesh sequence into a multi-resolution representation comprises the steps of:

setting a number of layers (L) for decomposition,

encoding the set number of layers (L) for the decomposition,

selecting one or more patches comprising sets of triangles having one middle vertex in common, the middle vertex having a valence not more than a predefined number, in particular not more than 6, the valence being defined by the number of direct neighbour vertices of said middle vertex,

removing the middle vertices of said one or more selected patches and generating a decomposed layer (V_L) defined by said removed middle vertices with associated connectivity (K_L),

determining a simplified connectivity associated to the remaining set of vertices,

repeating the steps of selecting one or more patches, removing the middle vertices, and determining a simplified connectivity for a number of iterations (L−

2) being by 2 less than the number of set layers on the respective remaining sets of vertices with associated simplified connectivities to generate decomposed layers (V_L-1, . . . , V₂) with associated connectivities (K_L-1, . . . , K₂), and

defining the remaining set of vertices (V₁) with associated connectivity (K₁) as a base layer.

67. The method according to claim 66, wherein the step of determining a simplified connectivity associated to the remaining set of vertices comprises the step of:

triangulating the vertices of the former 1-ring neighborhood of a removed vertex, wherein the 1-ring neighborhood is defined by all vertices being a direct neighbour of a vertex, wherein a triangulation is selected based on an average absolute deviation of the valences of the vertices of the 1-ring neighborhood of the removed vertex from a valence equal to a predefined number, in particular equal to 6, with respect to the triangulation and the connectivity (K_l) associated to the layer,

wherein the step of encoding the vertices of the layers using a predictive encoder comprises the step of;

selecting a compression method for each frame being selected of the group comprising of

an I-frame compression and

a P-frame compression and/or

a B-frame compression

and generating a prediction error using an I-frame predictor, a P-frame predictor, or a B-frame predictor respectively,

wherein a P-frame compression for a current vertex of a current frame, considering an already encoded frame as a reference frame comprises calculating a P-frame predictor by adding up an I-frame predictor of the current vertex of the current frame and the difference of the location of the current vertex in the reference frame and an I-frame predictor of the current vertex of the reference frame respectively,

whereby said difference is multiplied by a Rotation Matrix (A).

68. The method according to claim 64, wherein a B-frame compression for a current vertex of a current frame, considering two already encoded frames as a first reference frame and second reference frame respectively, comprises calculating a B-frame predictor by calculating an average, in particular a weighted average of a P-frame predictor according to claim 67 considering the first reference frame and a P-frame predictor according to claim 67 considering the second reference frame.

69. The method according to claim 67, wherein an information for specifying the chosen predictor is encoded whereby the predictor is encoded frame-wise or layer-wise, in particular using a fixed number of bits.

70. Encoding device for encoding a time-consistent 3D mesh sequence adapted for carrying out the method of claim 64.

74. The method according to any of the claim 71 or 72, wherein the method is adapted to decode a data signal being coded by a method according to claim 64.

76. Computer program for encoding a time-consistent 3D mesh sequence, the computer program comprising program code means for causing an encoding device as defined in claim 70 to carry out steps of the encoding method as defined in claim 64, when the computer program is run on a computer controlling the encoding device.

78. Data signal representing an encoded time-consistent 3D mesh sequence being encoded according to claim 64.

71. Method for decoding a data signal representing an encoded time-consistent 3D mesh sequence defined by frames, each frame comprises a set of vertices (V) with associated connectivity (K) and wherein the connectivity (K) is identical for all frames of the 3D mesh sequence, encoded by using scalable predictive coding, the method comprising the steps of:

receiving the data signal,

decoding the connectivity (K) of the 3D mesh sequence, decomposing the 3D mesh sequence into a multi-resolution representation comprising two or more spatial layers represented by disjoint sets of vertices (V_l) with associated connectivities (K_l) and

decoding the vertices of the layers using a predictive decoder,

wherein the decomposition is calculated from the connectivity (K) of the 3D mesh sequence irrespectively of location data of the vertices.

72. The method according to claim 71, further comprising the step:

decoding for each frame and/or for each layer information specifying a compression method as used by the coder, in particular decoding reference frame numbers representing the numbers of reference frames and decoding the type of compression method comprising I-frame compression and P-frame compression and/or B-frame compression,

wherein decomposing the 3D mesh sequence into a multi-resolution representation comprises

decoding a layer number representing the number of layers (L) as received from a coder,

selecting one or more patches comprising sets of triangles having one middle vertex in common, the middle vertex having a valence not more than a predefined number, in particular not more than 6, the valence being defined by the number of direct neighbour vertices of said middle vertex,

removing the middle vertices of said one or more selected patches and generating a decomposed layer (V_L) defined by said removed middle vertices with associated connectivity (K_L),

determining a simplified connectivity associated to the remaining set of vertices,

repeating the steps of selecting one or more patches, removing the middle vertices, and determining a simplified connectivity for a number of iterations (L−

2) being by 2 less than the number of set layers on the respective remaining sets of vertices with associated simplified connectivities to generate decomposed layers (V_L-1, . . . , V₂) with associated connectivities (K_L-1, . . . , K₂), and

defining the remaining set of vertices (V₁) with associated connectivity (K₁) as a base layer,

wherein the step of determining a simplified connectivity associated to the remaining set of vertices comprises the step of;

triangulating the vertices of the former 1-ring neighborhood of a removed vertex, wherein the 1-ring neighborhood is defined by all vertices being a direct neighbour of a vertex, wherein a triangulation is selected based on an average absolute deviation of the valences of the vertices of the 1-ring neighborhood of the removed vertex from a valence equal to a predefined number, in particular equal to 6, with respect to the triangulation and the connectivity (K_l) associated to the layer,

further comprising the steps of determining a predictor according to the specified compression method, decoding a prediction error and calculating a prediction value and/or the location data of the corresponding vertex based on the specified predictor and the decoded prediction error whereby the steps are performed preferably in each layer and

decoding the location data of the vertices of a base layer,

wherein in case of an I-frame compression for a current vertex of a current frame comprises calculating an I-frame predictor as an average of the locations of an already decoded 1-ring neighbourhood of said current vertex in said current frame, and wherein in case of a P-frame compression for a current vertex of a current frame the method comprises, considering an already decoded frame as a reference frame and calculating a P-frame predictor by adding up an I-frame predictor of the current vertex of the current frame calculated as in case of an I-frame compression and the difference of the location of the current vertex in the reference frame and an I-frame predictor of the current vertex of the reference frame calculated as in case of an I-frame compression respectively, and said difference is multiplied by a Rotation Matrix (A), and

wherein in case of a B-frame compression for a current vertex of a current frame, the method comprises considering two already decoded frames as a first reference frame and second reference frame respectively, and calculating a B-frame predictor by calculating an average, in particular a weighted average, of a P-frame predictor as in case of a P-frame compression considering the first reference frame and a P-frame predictor as in case of a P-frame compression considering the second reference frame.

73. The method according to any of claim 71 or 72, wherein an information for specifying the chosen predictor is decoded frame-wise or layer-wise, in particular using a fixed number of bits.

75. Decoding device for decoding a data signal representing an encoded time-consistent 3D mesh sequence adapted for carrying out the method of claim 71.

77. Computer program for decoding a data signal representing an encoded time-consistent 3D mesh sequence, the computer program comprising program code means for causing an decoding device as defined in claim 75 to carry out steps of the decoding method as defined in claim 71, when the computer program is run on a computer controlling the decoding device.