Apparatus and method for creating three-dimensional personalized figure

• US 9,846,804 B2
• Filed: 03/03/2015
• Issued: 12/19/2017
• Est. Priority Date: 03/04/2014
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Lim, Seong-Jae, Hwang, Bon-Woo, Kim, Kap-Kee, Yoon, Seung-Uk, Jun, Hye-Ryeong, Choi, Jin-Sung, Koo, Bon-Ki
• Primary Examiner(s): Woldemariam, Aklilu
• Application Number:
  US14/637,344
• Publication Number:
  US 20150254502A1
• Time in Patent Office: 1,022 Days
• Field of Search: 382154, 382285, 345419, 345420, 345441, 345646
• CPC Class Codes: B33Y 80/00 Products made by additive manufacturing
• CPC Class Codes: G05B 19/40931 concerning programming of geometry
• CPC Class Codes: G05B 2219/2636 Reproduction, image copying machine
• CPC Class Codes: G05B 2219/49023 3-D printing, layer of powder, add drops of binder in layer, new powder
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

Disclosed herein is an apparatus and method for automatically creating a 3D personalized figure suitable for 3D printing by detecting a face area and features for respective regions from face data acquired by heterogeneous sensors and by optimizing global/local transformation. The 3D personalized figure creation apparatus acquires face data of a user corresponding to a reconstruction target; extracts feature points for respective regions from the face data, and reconstructs unique 3D models of the user'"'"'s face, based on the extracted feature points; creates 3D figure models based on the unique 3D models and previously stored facial expression models and body/adornment models; and verifies whether each 3D figure model has a structure and a shape corresponding to actual 3D printing, corrects and edits the 3D figure model based on results of verification, and outputs a 3D figure model corresponding to 3D printing.

Claims

1. A method for creating a three-dimensional (3D) personalized figure, the method being performed by an apparatus for creating a 3D personalized figure, comprising:

acquiring face data captured by a plurality of heterogeneous sensors of a user corresponding to a reconstruction target;

extracting feature points for respective regions from the face data, and reconstructing unique 3D models of the user'"'"'s face, based on the extracted feature points;

creating 3D figure models based on the unique 3D models and previously stored facial expression models and body/adornment models by;

generating facial expressions that match an actually input facial expression of the user, based on the unique 3D models and the previously stored facial expression models; and

selecting a facial expression that stochastically matches a facial expression of the actual user from among the created facial expressions, and creating the 3D figure models based on the selected facial expression;

verifying whether each 3D figure model has a structure and a shape corresponding to actual 3D printing;

correcting and editing the 3D figure model based on results of the verification; and

outputting a 3D figure model corresponding to 3D printing,

wherein reconstructing the unique 3D models of the user'"'"'s face comprises;

generating a front image based on the face data;

detecting a face area from the front image;

detecting feature points for respective regions from the face area;

detecting 3D corresponding points of the face data corresponding to the feature points of a 3D standard model, based on the feature points for respective regions, and then matching and transforming appearance information of the 3D standard model;

generating a face texture map using the transformed appearance information of the 3D standard model and the front image; and

creating the unique 3D models of the user'"'"'s face using the face texture map.

2. The method of claim 1, the unique 3D models of the user'"'"'s face comprises creating the unique 3D models of the user'"'"'s face based on a procedure of causing a color of a remaining area of the face, which is not captured, to match a color of the face area, which is captured, using the face texture map.

3. The method of claim 1, wherein creating the 3D figure models comprises:

creating adorned unique 3D models by combining the previously stored body/adornment models with the unique 3D models; and

selecting a body/adornment model that stochastically matches that of the actual user from among the adorned unique 3D models.

4. The method of claim 1, wherein the face data corresponds to a 3D unrefined mesh model.

5. An apparatus for creating a 3D personalized figure, comprising:

an information acquisition unit for acquiring face data of a user corresponding to a reconstruction target from a plurality of heterogeneous sensors;

a face reconstruction unit for extracting feature points for respective regions from the face data, and reconstructing unique 3D models of the user'"'"'s face, based on the extracted feature points;

a model creation unit for creating 3D figure models based on the unique 3D models and previously stored facial expression models and body/adornment models; and

a facial expression generation unit for generating facial expressions that match an actually input facial expression of the user, based on the unique 3D models and the previously stored facial expression models;

a selection unit for selecting a facial expression that stochastically matches that of the actual user from among the created facial expressions, and creates the 3D figure models based on the results of selection;

a model verification unit for verifying whether each 3D figure model has a structure and a shape corresponding to actual 3D printing, correcting and editing the 3D figure model based on results of verification, and outputting a 3D figure model corresponding to 3D printing,

wherein the face reconstruction unit comprises;

a generation unit for generating a front image based on the face data;

a face area detection unit for detecting a face area from the front image;

a feature point detection unit for detecting feature points for respective regions from the face area;

a matching and transformation unit for detecting 3D corresponding points of the face data corresponding to the feature points of a 3D standard model, based on the feature points for respective regions, and then matching and transforming appearance information of the 3D standard model;

a texture map generation unit for generating a face texture map using the transformed appearance information of the 3D standard model and the front image; and

a unique 3D model creation unit for creating the unique 3D models of the user'"'"'s face using the face texture map.

6. The apparatus of claim 5, wherein the face data corresponds to a 3D unrefined mesh model.

7. The apparatus of claim 5, wherein the unique 3D model creation unit creates the unique 3D models of the user'"'"'s face based on a procedure of causing a color of a remaining area of the face, which is not captured, to match a color of the face area, which is captured, using the face texture map.

8. The apparatus of claim 5, wherein the model creation unit comprises:

a body/adornment model creation unit for creating adorned unique 3D models by combining the previously stored body/adornment models with the unique 3D models;

wherein the selection unit selects a body/adornment model that stochastically matches that of the actual user from among the adorned unique 3D models.

9. The apparatus of claim 8, wherein the model creation unit is configured such that a facial expression model storage unit including the facial expression models and a body/adornment model storage unit including the body/adornment models are operated in conjunction with each other.