System and method for tracking facial muscle and eye motion for computer graphics animation

• US 7,554,549 B2
• Filed: 11/08/2004
• Issued: 06/30/2009
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Pictures Entertainment Incorporated (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Pictures Entertainment Incorporated (Sony Group Corp.)
• Inventors: Sagar, Mark, Scott, Remington
• Primary Examiner(s): Tung; Kee M
• Assistant Examiner(s): MATHEWS, CRYSTAL ANN
• Application Number:
  US10/984,488
• Publication Number:
  US 20060071934A1
• Time in Patent Office: 1,695 Days
• Field of Search: 600/546, 345473-474
• US Class Current: 345/473
• CPC Class Codes: A61B 2562/04 Arrangements of multiple sensors of the same type
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/1126 using a particular sensing technique
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

A motion tracking system enables faithful capture of subtle facial and eye motion using a surface electromyography (EMG) detection method to detect muscle movements and an electrooculogram (EOG) detection method to detect eye movements. An embodiment of the motion tracking animation system comprises a plurality of pairs of EOG electrodes adapted to be affixed to the skin surface of the performer at locations adjacent to the performer'"'"'s eyes. The EOG data comprises electrical signals corresponding to eye movements of a performer during a performance. Programming instructions further provide processing of the EOG data and mapping of processed EOG data onto an animated character. As a result, the animated character will exhibit he same muscle and eye movements as the performer.

Claims

1. A motion tracking animation system, comprising:

a plurality of pairs of electromyography (EMG) electrodes adapted to be affixed to a skin surface of a performer at plural locations corresponding to respective muscles, wherein at least one pair of EMG electrodes is aligned along an expected direction of an electrical signal corresponding to muscle movements of the performer when the electrical signal is to be read, and at least one pair of EMG electrodes is aligned along a direction perpendicular to the expected direction of the electrical signal when the electrical signal is to be excluded;

a processor operatively coupled to said plurality of pairs of EMG electrodes, said processor including programming instructions to perform the functions of;

acquiring EMG data from said plurality of pairs of EMG electrodes, said EMG data comprising electrical signals corresponding to muscle movements of the performer during a performance;

processing the EMG data to provide a digital model of the muscle movements, wherein the digital model further comprises a virtual facial muscle structure; and

mapping the digital model onto an animated character, the animated character will exhibit the same muscle movements as the performer when corresponding muscle vectors within the virtual facial muscle structure are actuated in response to the acquired EMG data; and

a plurality of pairs of electrooculogram (EOG) electrodes adapted to be affixed to said skin surface of the performer at locations adjacent to the performer'"'"'s eyes, said processor operatively coupled to said plurality of pairs of EOG electrodes and including programming instructions to perform the functions of;

acquiring EOG data from said plurality of pairs of EOG electrodes, said EOG data comprising electrical signals corresponding to orientations of the performer'"'"'s eyes during a performance;

processing the EOG data by determining a magnitude and a polarity of a dipole electric field associated with the performer'"'"'s eyes in order to reconstruct the orientations of the eyes; and

mapping the processed EOG data onto the animated character, wherein the animated character will exhibit the same eye orientations as the performer when corresponding muscle vectors within the virtual facial muscle structure are actuated in response to the acquired EOG data.

2. The system of claim 1, wherein said processing function further includes generating a set of calibration poses based in part on the processed EMG data.

3. The system of claim 1, wherein said acquiring EMG data function further includes rectifying and/or filtering the electrical signals.

4. The system of claim 1, wherein said processor further comprises a computer graphics animation processor.

5. The system of claim 1, further comprising an electrode interface coupled to said plurality of EMG electrodes, said electrode interface providing a composite signal to said processor.

6. The system of claim 5, wherein said electrode interface is adapted to be carried on a body of the performer.

7. The system of claim 5, wherein said electrode interface has a wireless connection to said processor.

8. A method of animating using motion tracking, comprising:

affixing a plurality of pairs of electromyography (EMG) electrodes to a skin surface of a performer at plural locations corresponding to respective muscles, wherein at least one pair of EMG electrodes is aligned along an expected direction of an electrical signal corresponding to muscle movements of the performer when the electrical signal is to be read, and at least one pair of EMG electrodes is aligned along a direction perpendicular to the expected direction of the electrical signal when the electrical signal is to be excluded;

acquiring EMG data from said plurality of pairs of EMG electrodes, said EMG data comprising electrical signals corresponding to muscle movements of the performer during a performance;

processing the EMG data to provide a digital model of the muscle movements, wherein the digital model further comprises a virtual facial muscle structure;

mapping the digital model onto an animated character, wherein the animated character will exhibit the same muscle movements as the performer when corresponding muscle vectors within the virtual facial muscle structure are actuated in response to the acquired EMG data;

affixing a plurality of pairs of electooculogram (EOG) electrodes to said skin surface of the performer at locations adjacent to the performer'"'"'s eyes; and

acquiring EOG data from said plurality of pairs of EOG electrodes, said EOG data comprising electrical signals corresponding to orientations of the performer'"'"'s eyes during a performance, processing the EOG data by determining a magnitude and a polarity of a dipole electric field associated with the performer'"'"'s eyes in order to reconstruct the orientations of the eyes, and mapping the processed EOG data onto the animated character, wherein the animated character will exhibit the same eye orientations as the performer when corresponding muscle vectors within the virtual facial muscle structure are actuated in response to the acquired EMG data.

9. The method of claim 8, wherein said processing step further includes generating a set of calibration poses based in part on the processed EMG data.

10. The method of claim 8, wherein said acquiring EMG data step further includes rectifying and/or filtering the electrical signals.

11. The method of claim 8, further comprising carrying an electrode interface operatively coupled to said plurality of EMG electrodes on a body of the performer.