Process for producing and controlling animated facial representations

• US 6,020,892 A
• Filed: 09/29/1997
• Issued: 02/01/2000
• Est. Priority Date: 04/17/1995
• Status: Expired due to Fees

• Current Assignee: Kelly Dillon
• Original Assignee: Kelly Dillon
• Inventors: Dillon, Kelly
• Primary Examiner(s): Hong, Stephen S.
• Assistant Examiner(s): Kindred, Alford W.
• Application Number:
  US08/939,387
• Time in Patent Office: 855 Days
• Field of Search: 345/419, 345/429, 345/428, 345/116, 345/473, 345/132, 345/136, 345/133, 345/127, 345/121
• US Class Current: 345/419
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

A process for real-time motion control of ultra-high quality animated faces of any kind (human, dog, fish, etc.), or control of robotic faces. The system uses common motion capture equipment to develop high and low resolution data sets of facial expressions for human actor faces (serving as the input devices). The high-resolution data is manually and semi-automatically enhanced to provide additional information pertaining to any given expression. The high-resolution expression data sets are logically linked to robotic and computer model faces of any shape for discrete control of all robotic motion controls and the creation of a high resolution data set of model expressions. Real-time low-resolution motion capture data is matched to the low-resolution data sets, and thereby matched to the enhanced high resolution model data sets. Real-time model motion and robotic motion is thus effected by the real time motion capture, though the motion is made more real and fluid by the additional pre-processed information provided in the high resolution index.

Claims

1. A process for producing and controlling animation of Computer Model Faces and Robotic Faces, each of said Robotic Faces having a plurality of position controllers, said process being implemented with a High Resolution Capture System, a Low Resolution Motion Capture System, a plurality of High Density Markers, a plurality of actors, each having a face providing facial expressions including wrinkles, skin tone, lips and eyelids and Model Nodes, said process comprising the steps of:

A) preparing a High Resolution Index using

(1) said High Resolution Capture System using said High Density Markers on said face of one of said actors, capturing high resolution information for a variety of High Resolution Index facial expressions, said High Resolution Index comprising said high resolution facial expressions including geometric information about each of said expressions, and

(2) adding additional information to each of said high resolution expressions including information about said wrinkles, skin tone, lips and eyelids;

B) preparing a Low Resolution Index for each actor of said plurality of actors to associate each actor to said High Resolution Index by;

(1) using said Low Resolution Tracking System with each actor of said plurality of actors to capture low resolution facial expressions equivalent to the high resolution expressions in the High Resolution Index, said Low Resolution Index comprising said low resolution expressions,

(2) processing said Low Resolution Index expressions into facial segments and processing said segments to be identifiable as unique, and

(3) linking said Low Resolution Index expressions to said High Resolution Index expressions by numerically associating them on a one-to-one basis;

C) preparing Computer Model Faces including a Computer Model Neutral Face, said Computer Model Faces each having a plurality of vertex points;

D) associating said Computer Model Faces to which Index by

(1) placing said Model Nodes on said Computer Model Neutral Face,

(2) defining the motions of said Model Nodes for correlation of motions of specific ones of said High Density Markers in said facial expressions in said High Resolution Index, said correlations including variation of motion direction and non-linearity in direction and magnitude of said motions;

(3) logically constraining said Computer Model Face from undesired motion by controlling, including preventing, motions of said Model Nodes in certain directions, and

(4) processing said vertex points of said Computer Model Face with respect to said Nodes on said face by identifying each of said vertex point'"'"'s Model Nodes of influence;

E) creating a Model Index including Model Index expressions by

(1) using said Computer Model Neutral Face, information for each of said expressions of said High Resolution Index and information of said association of said Computer Model Neutral Face to said High Resolution Index, processing a unique Computer Model Face correlating to each of said expressions in said High Resolution Index, said Model Index comprising said Computer Model Faces;

(2) linking said expressions of said Model Index to said expressions of said High Resolution Index by numerically associating them on a one-to-one basis;

F) capturing and processing real time facial motion of one of said plurality of actors in low resolution to produce a shot, said shot comprising expressions in real time, or any time less than real time, said expressions being captured using said Low Resolution Tracking System, and

(1) processing the real-time facial motion expressions in facial segments, each of said expressions having a unique low resolution signature,

(2) matching each of said real-time facial motion expression segments by weighted value to at least one of said expression segments of said Low Resolution Index facial expressions, said expressions being closely similar, and

(3) combining the motions of said segments in the Model Index to form a unique model expressions for each of said shot expressions;

G) creating real-time Computer Model expressions by;

(1) moving points on said Computer Model Neutral Face through time according to the motions of each facial expression in said shot, and

H) moving said Computer Model or said Robotic Face by moving said Nodal Points of said Computer Model or said positional controllers in real time according to specific motions of said Nodal Points on said Model Face.

2. The process of claim 1 in which said High Resolution Capture System is implemented by a plurality of High Density Markers, the face of a human actor, a plane of reference, at least two cameras and computer processing equipment and comprising the steps of:

A) placing said High Density Markers on said face,

B) recording a plurality of expressions on said face with said at least two cameras to produce a High Resolution Index,

C) using said computer processors, processing said High Resolution Index to resolve the positions of said High Density Markers in three-dimension, and

D) using said computer processors, mathematically scaled, translating and rotating each of said expressions to a common scale and normal to said given plane of reference.

3. The process of claim 1 in which said Low Resolution Capture System is implemented with two digital cameras, the face of a human actor, a plurality of Low Density Markers and computer processing equipment and comprising the steps of:

A) placing said Low Density Markers on said face,

B) using said cameras, recording facial expressions made on said face, said recorded expressions comprising a Low Resolution Index,

C) processing said Low Resolution Index to resolve in three-dimensions the positions of said Low Density Markers,

D) dividing said recorded face into logical segments, and

E) mathematically scaling, translating and rotating each segment of each expression in said Low Resolution Index as needed to a common scale and in reference to a given plane.