THREE DIMENSIONAL CONTENT GENERATING APPARATUS AND THREE DIMENSIONAL CONTENT GENERATING METHOD THEREOF

• US 20170018088A1
• Filed: 06/17/2016
• Published: 01/19/2017
• Est. Priority Date: 07/14/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: LIMONOV, Alexander, JEONG, Jae-yun
• Granted Patent: US 10,269,175 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A three dimensional (3D) content generating apparatus includes an inputter configured to receive a plurality of images of an object captured from different locations; a detector configured to identify the object and detect a predetermined feature point of the object from each of the plurality of images; a map former configured to extract 3D location information of the detected feature point, and configured to form at least one depth map with respect to a surface of the object based on the extracted 3D location information of the feature point; and a content generator configured to generate a 3D content of the object using the at least one depth map and the plurality of images.

Claims

1. A three dimensional (3D) content generating apparatus comprising:

an inputter configured to receive a plurality of images of an object, the plurality of images being captured from different locations;

a detector configured to identify the object and detect a feature point of the object from each of the plurality of images;

a map former configured to extract 3D location information of the detected feature point, and configured to form at least one depth map with respect to a surface of the object based on the extracted 3D location information of the feature point; and

a content generator configured to generate a 3D content of the object based on the at least one depth map and the plurality of images.

2. The apparatus according to claim 1, further comprising a location determiner configured to determine a capturing location, from which each of the plurality of images is captured, based on the extracted 3D location information of the feature point,

wherein the content generator is configured to generate the 3D content of the object based on the determined capturing location.

3. The apparatus according to claim 1, wherein the detector is configured to apply a learned model of the object to the plurality of images and detect the feature point of the object.

4. The apparatus according to claim 3, wherein the detector is configured to update the model of the object by using at least one of the plurality of images, the determined capturing location, and the detected feature point, and is configured to detect the feature point by applying the updated model to an image subsequently input.

5. The apparatus according to claim 1, wherein the content generator is configured to select an image, from among the plurality of images, based on at least one of an arrangement shape of the detected feature point and the determined capturing location, and is configured to restore, by using the selected image, a texture of at least one predetermined part of the object, to generate the 3D content.

6. The apparatus according to claim 5, wherein the content generator is configured to select at least two images, from among the plurality of images, based on the at least one of the arrangement shape of the detected feature point and the determined capturing location, and is configured to restore, by using the at least two selected images, a texture of a part other than the predetermined part, to generate the 3D content.

7. The apparatus according to claim 2, wherein the map former is configured to:

select an image, from among the plurality of images, based on at least one of a distribution of the detected feature point and the determined capturing location; and

form the depth map based on depth information extracted from the selected image, and

the content generator is configured to;

determine a volume of a voxel cube, the volume being equal to or greater than the object;

rotate the voxel cube based on a learned model of the object; and

apply a higher resolution to at least one predetermined part of the object than to a part other than the predetermined part.

8. The apparatus according to claim 2, wherein the map former is configured to:

form the depth map through stereo processing between at least two images among the plurality of images, and

determine a size of a patch for the stereo processing based on at least one of the 3D location information of the feature point detected from the at least two images and capturing locations from which the at least two images are captured.

9. The apparatus according to claim 1, wherein the map former is configured to determine a depth range with respect to the surface of the object based on the 3D location information of the feature point, and is configured to form the depth map based on the determined depth range.

10. The apparatus according to claim 1, wherein the content generator is configured to determine a volume of the object based on the 3D location information of the feature point, and is configured to generate the 3D content based on the determined volume.

11. The apparatus according to claim 1, wherein the plurality of images are obtained by capturing the object by using a monocular camera at the different locations.

12. A 3D content generating method comprising:

receiving an input of a plurality of images of an object, the plurality of images being captured from different locations;

identifying the object and detecting a feature point of the object from each of the plurality of images;

extracting 3D location information of the detected feature point, and forming at least one depth map with respect to a surface of the object based on the extracted 3D location information of the feature point; and

generating a 3D content of the object based on the at least one depth map and the plurality of images.

13. The method according to claim 12, further comprising determining a capturing location, from which each of the plurality of images is captured, based on the 3D location information of the feature point,

wherein the generating comprises generating the 3D content of the object based on the determined capturing location.

14. The method according to claim 12, wherein the detecting comprises applying a learned model of the object to the plurality of images to detect the feature point of the object.

15. The method according to claim 14, wherein the detecting comprises:

updating the model of the object by using at least one of the plurality of images, the determined capturing location, and the detected feature point; and

detecting the feature point by applying the updated model to an image subsequently input.

16. The method according to claim 12, wherein the generating comprises selecting an image, from among the plurality of images, based on at least one of an arrangement shape of the detected feature point and the determined capturing location, and restoring, by using the selected image, a texture of at least one predetermined part of the object.

17. The method according to claim 16, wherein the generating comprises selecting at least two images, from among the plurality of images, based on the at least one of the arrangement shape of the detected feature point and the determined capturing location, and restoring, by using the at least two selected images, a texture of a part other than the predetermined part.

18. The method according to claim 13, wherein the forming the depth map comprises:

selecting an image, from among the plurality of images, based on at least one of a distribution of the detected feature point and the determined capturing location;

forming the depth map based on depth information extracted from the selected image;

determining a volume of a voxel cube, the volume being equal to or greater than the object;

rotating the voxel cube based on a learned model of the object; and

applying a higher resolution to at least one predetermined part of the object than to a part other than the predetermined part.

19. The method according to claim 13, wherein the forming the depth map comprises forming the depth map through stereo processing between at least two images among the plurality of images, and

determining a size of a patch for the stereo processing based on at least one of the 3D location information of the feature point detected from the at least two images and capturing locations from which the at least two images are captured.

20. A non-transitory record medium that stores a program which, when executed by a computer, causes the computer to execute a 3D content producing method, the 3D content generating method comprising:

receiving an input of a plurality of images of an object, the plurality of images being captured from different locations;

identifying the object and detecting a feature point of the object from each of the plurality of images;

extracting 3D location information of the detected feature point, and forming at least one depth map with respect to a surface of the object based on the extracted 3D location information of the feature point; and

generating a 3D content of the object based on the at least one depth map and the plurality of images.