3-dimensional shape reconstruction device using depth image and color image and the method

• US 9,384,585 B2
• Filed: 10/16/2013
• Issued: 07/05/2016
• Est. Priority Date: 10/23/2012
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Ho Won, Choi, Jin Sung, Koo, Bon Ki
• Primary Examiner(s): Hon, Ming
• Assistant Examiner(s): Doan, Phuc
• Application Number:
  US14/055,257
• Publication Number:
  US 20140111507A1
• Time in Patent Office: 993 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation

Abstract

A 3-dimensional (3D) whole shape reconstruction apparatus using a depth image and a color image and a 3D whole shape reconstruction method are provided. The 3D whole shape reconstruction apparatus may include an information generation unit to generate shape information of 3D portions using a depth image and a color image, and a shape reconstruction unit to reconstruct a 3D whole shape of a user by matching the shape information of 3D portions.

Claims

1. A 3-dimensional (3D) whole shape reconstruction apparatus comprising:

at least a computer configured to;

generate shape information of 3D portions based on inner silhouette information of a user using a depth image and outer silhouette information of the user using a color image;

reconstruct a 3D whole shape of the user by matching the shape information of 3D portions; and

correct or add depth information generated at a silhouette boundary of the depth image, by;

generating an alpha mask image with respect to a foreground image including the user, using a color difference between the foreground image and the background image,

generating a geometrically corrected depth image by projecting the depth image to the color image using geometric correction information of the depth image, and

obtaining the depth information on the geometrically corrected depth image corresponding to a hole of the alpha mask image by combining the foreground image with the alpha mask image.

2. The 3D whole shape reconstruction apparatus of claim 1, wherein the computer is further configured to use a background image to generate shape information of the 3D portions.

3. The 3D whole shape reconstruction apparatus of claim 1, wherein the computer is further configured to generate the shape information of 3D portions considering characteristics of the depth information and characteristics of the outer silhouette information of the user, provided by the color image, using a background image.

4. The 3D whole shape reconstruction apparatus of claim 1, wherein the computer is further configured to extract a particular pixel of the color image not including depth information, and to estimates depth information corresponding to the particular pixel.

5. The 3D whole shape reconstruction apparatus of claim 1, wherein the computer is further configured to generate a color edge image and a color boundary image of the user based on the color image.

6. The 3D whole shape reconstruction apparatus of claim 1, wherein the computer is further configured to reconstruct the 3D whole shape using shape patch information generated based on the shape information of 3D portions.

7. The 3D whole shape reconstruction apparatus of claim 6, wherein the shape patch information is generated by converting a local coordinate expressing the shape information of 3D portions into a global coordinate.

8. The 3D whole shape reconstruction apparatus of claim 6, wherein the computer is further configured to reconstruct the 3D whole shape by elliptically modeling the shape patch information and matching the shape patch information with respect to an elliptical model of particular shape patch information according to a movement of the user.

9. A 3-dimensional (3D) whole shape reconstruction method comprising:

generating shape information of 3D portions based on inner silhouette information of a user using a depth image and outer silhouette information of the user using a color image;

reconstructing a 3D whole shape of the user by matching the shape information of 3D portions;

correcting or adding depth information generated at a silhouette boundary of the depth image; and

wherein the correcting or adding of the depth information comprises;

generating an alpha mask image with respect to a foreground image including the user, using a color difference between the foreground image and the background image,

generating a geometrically corrected depth image by projecting the depth image to the color image using geometric correction information of the depth image, and

obtaining the depth information on the geometrically corrected depth image corresponding to a hole of the alpha mask image by combining the foreground image with the alpha mask image.

10. The 3D whole shape reconstruction method of claim 9, wherein the generating comprises using a background image to generate shape information of the 3D portions.

11. The 3D whole shape reconstruction method of claim 9, wherein the generating comprises generating the shape information of 3D portions considering characteristics of the depth information and characteristics of the outer silhouette information of the user, provided by the color image, using a background image.

12. The 3D whole shape reconstruction method of claim 9, wherein the generating comprises extracting a particular pixel of the color image not including depth information of the depth image, and estimating depth information corresponding to the particular pixel.

13. The 3D whole shape reconstruction method of claim 9, wherein the generating comprises generating a color edge image and a color boundary image of the user based on the color image.

14. The 3D whole shape reconstruction method of claim 9, wherein the reconstructing comprises reconstructing the 3D whole shape using shape patch information generated based on the shape information of 3D portions.

15. The 3D whole shape reconstruction method of claim 14, wherein the shape patch information is generated by converting a local coordinate expressing the shape information of 3D portions into a global coordinate.

16. The 3D whole shape reconstruction method of claim 14, wherein the reconstructing comprises reconstructing

the 3D whole shape by elliptically modeling the shape patch information and matching the shape patch information with respect to an elliptical model of particular shape patch information according to a movement of the user.