Three dimensional content generating apparatus and three dimensional content generating method thereof

• US 10,269,175 B2
• Filed: 06/17/2016
• Issued: 04/23/2019
• Est. Priority Date: 07/14/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jeong, Jae-yun, Limonov, Alexander
• Primary Examiner(s): Cese, Kenny A
• Application Number:
  US15/185,896
• Publication Number:
  US 20170018088A1
• Time in Patent Office: 1,040 Days
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A three dimensional (3D) content generating apparatus includes an inputter configured to receive a plurality of images of an object captured from different locations; a detector configured to identify the object and detect a predetermined feature point of the object from each of the plurality of images; a map former configured to extract 3D location information of the detected feature point, and configured to form at least one depth map with respect to a surface of the object based on the extracted 3D location information of the feature point; and a content generator configured to generate a 3D content of the object using the at least one depth map and the plurality of images.

Claims

1. An electronic apparatus comprising:

a memory storing instructions;

a processor configured to execute the stored instructions to;

receive a plurality of images of an object, the plurality of images being captured from a plurality of capturing locations,

detect at least one feature point of the object from each of the plurality of images by applying a trained artificial intelligence (AI) model of the object to the plurality of images,

obtain 3D location information of the detected at least one feature point by inputting the detected at least one feature point to the trained AI model,

form at least one depth map with respect to a surface of the object based on the obtained 3D location information,

determine the plurality of capturing locations, from which each of the plurality of images is captured, based on the extracted 3D location information of the feature point, and

generate a 3D content of the object based on the plurality of capturing locations, the at least one depth map and at least one image of the plurality of images,

wherein the processor is configured to update the trained AI model by using at least one of the plurality of images, the determined plurality of capturing locations, and the detected at least one feature point.

2. The electronic apparatus according to claim 1, wherein the processor is configured to detect a feature point in an image subsequently input by applying the updated trained AI model to the image subsequently input.

3. The electronic apparatus according to claim 1, wherein the processor is configured to select an image, from among the plurality of images, based on an arrangement shape of the detected at least one feature point and the determined plurality of capturing locations, and is configured to restore, by using the selected image, a texture of at least one predetermined part of the object, to generate the 3D content.

4. The electronic apparatus according to claim 3, wherein the processor is configured to select at least two images, from among the plurality of images, based on the arrangement shape of the detected at least one feature point and the determined plurality of capturing locations, and is configured to restore, by using the at least two selected images, a texture of a part other than the at least one predetermined part, to generate the 3D content.

5. The electronic apparatus according to claim 1, wherein the processor is configured to:

select an image, from among the plurality of images, based on at least one of a distribution of the detected at least one feature point and the determined plurality of capturing locations; and

form the at least one depth map based on depth information extracted from the selected image, and

the processor is further configured to;

determine a volume of a voxel cube, the volume being equal to or greater than the object;

rotate the voxel cube based on a learned model of the object; and

apply a higher resolution to at least one predetermined part of the object than to a part other than the predetermined part.

6. The electronic apparatus according to claim 1, wherein the processor is configured to:

form the at least one depth map through stereo processing between at least two images among the plurality of images, and

determine a size of a patch for the stereo processing based on at least one of the 3D location information of the at least one feature point detected from the at least two images and the plurality of capturing locations from which the at least two images are captured.

7. The electronic apparatus according to claim 1, wherein the processor is configured to determine a depth range with respect to the surface of the object based on the 3D location information of the at least one feature point, and is configured to form the at least one depth map based on the determined depth range.

8. The electronic apparatus according to claim 1, wherein the processor is configured to determine a volume of the object based on the 3D location information of the at least one feature point, and is configured to generate the 3D content based on the determined volume.

9. The electronic apparatus according to claim 1, wherein the plurality of images are obtained by capturing the object by using a monocular camera at the plurality of capturing locations.

10. A method for generating a 3D content, the method comprising:

receiving an input of a plurality of images of an object, the plurality of images being captured from a plurality of capturing locations;

detecting at least one feature point of the object from each of the plurality of images by applying a trained artificial intelligence (AI) model of the object to the plurality of images;

obtaining 3D location information of the detected at least one feature point by inputting the detected at least one feature point to the trained AI model,

forming at least one depth map with respect to a surface of the object based on the obtained 3D location information of the feature point;

determining the plurality of capturing locations, from which each of the plurality of images is captured, based on the 3D location information of the feature point; and

generating a 3D content of the object based on the determined plurality of capturing locations, the at least one depth map and at least one image of the plurality of images,

wherein the trained AI model of the object is updated by using at least one of the plurality of images, the determined plurality of capturing locations, and the detected at least one feature point.

11. The method according to claim 10, wherein the detecting comprises

detecting a feature point of an image subsequently input by applying the updated model to the image subsequently input.

12. The method according to claim 10, wherein the generating comprises selecting an image, from among the plurality of images, based on an arrangement shape of the detected at least one feature point and the determined plurality of capturing locations, and restoring, by using the selected image, a texture of at least one predetermined part of the object.

13. The method according to claim 12, wherein the generating comprises selecting at least two images, from among the plurality of images, based on the arrangement shape of the detected at least one feature point and the determined plurality of capturing locations, and restoring, by using the at least two selected images, a texture of a part other than the predetermined part.

14. The method according to claim 10, wherein the forming the at least one depth map comprises:

selecting an image, from among the plurality of images, based on at least one of a distribution of the detected at least one feature point and the determined plurality of capturing locations;

forming the at least one depth map based on depth information extracted from the selected image;

determining a volume of a voxel cube, the volume being equal to or greater than the object;

rotating the voxel cube based on a learned model of the object; and

applying a higher resolution to at least one predetermined part of the object than to a part other than the predetermined part.

15. The method according to claim 10, wherein the forming the at least one depth map comprises forming the at least one depth map through stereo processing between at least two images among the plurality of images, and

determining a size of a patch for the stereo processing based on at least one of the 3D location information of the at least one feature point detected from the at least two images and the plurality of capturing locations from which the at least two images are captured.

16. A non-transitory record medium that stores a program which, when executed by a computer, causes the computer to execute a 3D content generating method, the 3D content generating method comprising:

receiving an input of a plurality of images of an object, the plurality of images being captured from a plurality of capturing locations;

detecting at least one feature point of the object from each of the plurality of images by applying a trained artificial intelligence (AI) model of the object to the plurality of images;

obtaining 3D location information of the detected at least one feature point by inputting the detected at least one feature point to the trained AI model,

forming at least one depth map with respect to a surface of the object based on the obtained 3D location information of the feature point;

determining the plurality of capturing locations, from which each of the plurality of images is captured, based on the 3D location information of the feature point; and

generating a 3D content of the object based on the determined plurality of capturing locations, the at least one depth map and at least one image of the plurality of images,

wherein the trained AI model of the object is updated by using at least one of the plurality of images, the determined plurality of capturing locations, and the detected at least one feature point.