Method and apparatus of facial image conversion by interpolation/extrapolation for plurality of facial expression components representing facial image

• US 6,088,040 A
• Filed: 08/27/1997
• Issued: 07/11/2000
• Est. Priority Date: 09/17/1996
• Status: Expired due to Fees

• Current Assignee: ATR Human Information Processing Research Laboratories
• Original Assignee: ATR Human Information Processing Research Laboratories
• Inventors: Oda, Masaomi, Chandrasiri, Naiwala Pathirannehelage
• Primary Examiner(s): Zimmerman, Mark K.
• Assistant Examiner(s): Nguyen, Kimbinh T.
• Application Number:
  US08/917,926
• Time in Patent Office: 1,049 Days
• Field of Search: 345/441, 345/435, 345/473, 345/425, 382/118, 382/224
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites

Abstract

An input/output control mechanism receives facial image information of different facial expressions of a plurality of human beings to provide control so that a normalization processing mechanism normalizes each facial image information. A facial expression generation mechanism converts the normalized facial image information into facial expression components, and averages the facial expression components for each facial expression to store the same as average facial expression components. The input/output control mechanism has the facial expression component converted by the facial expression generation mechanism after the applied facial image information is normalized. The facial expression generation mechanism converts the facial expression components of the applied facial image information according to a difference between an average facial expression component corresponding to a facial expression of applied facial image information and an average facial expression component of a specified facial expression. Synthesized facial image information is generated from the converted synthesized facial expression component.

Claims

1. An image information conversion method comprising the steps of:

extracting, from facial image information of a plurality of one-dimensional different facial expressions of a predetermined human being, a plurality of facial expression components for each said facial expression corresponding to respective projective components for a plurality of coordinate axes on facial image representation space taking the entire face as a processing unit,

calculating a plurality of synthesized facial components corresponding to facial image information of a synthesized facial expression at an arbitrary rate for specified arbitrary two of said different facial expressions by interpolation/extrapolation of a facial expression component corresponding to said two different facial expressions, and

generating synthesized facial image information according to said plurality of synthesized facial expression components.

2. The facial image information conversion method according to claim 1, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by Karhunen-Loeve Transform.

3. The facial image information conversion method according to claim 1, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by discrete cosine transform.

4. A facial image information conversion method comprising the steps of:

producing an average facial expression component difference between each facial expression by an average operation for a plurality of human beings according to facial expression components extracted from facial image information of a plurality of different facial expressions,

deforming arbitrary applied facial image information so that a plurality of reference points establish a predetermined position relationship with the entire face taken as a processing unit to convert the same into normalization facial image information, extracting a plurality of one-dimensional facial expression components corresponding to respective projective components for a plurality of coordinate axes on facial image representation space of said normalization facial image information,

calculating a plurality of synthesized facial expression components corresponding to a specified facial expression representation rate on the basis of an average facial expression component difference between each facial expression and said facial expression component corresponding to said applied facial image information, according to a facial expression of said applied facial image information and a specified facial expression,

generating normalization synthesized facial image information according to said plurality of synthesized facial expression components, and

deforming said normalization synthesized facial image information so that arrangement of reference points thereof are identical to the arrangement of reference points for original said applied facial image information to generate synthesized facial image information.

5. The facial image information conversion method according to claim 4, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by Karhunen-Loeve Transform.

6. The facial image information conversion method according to claim 4, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by discrete cosine transform.

7. A facial image information conversion method comprising the steps of:

specifying a plurality of reference points for each reference face image information with respect to a plurality of reference face image information of a different facial expression for a plurality of human beings,

deforming said reference face image information so that said plurality of reference points established a predetermined position relationship with the entire face taken as a processing unit for every said reference face image information and converting the same into normalization reference facial image information,

extracting a plurality of one-dimensional facial expression components corresponding to respective projective components for a plurality of coordinate axes on facial image representation space of each said normalization reference facial image information, and obtaining an average facial expression component for each said facial expression component with respect to said plurality of human beings for each facial expression and said average facial expression component difference between facial expressions,

specifying a plurality of reference points for arbitrary applied facial image information with the entire face taken as a processing unit,

deforming said applied facial image information so that said plurality of reference points establish a predetermined position relationship with the entire face taken as a processing unit, and converting the same into normalization facial image information,

extracting a plurality of facial expression components corresponding to respective projective components for a plurality of coordinate axes on image representation space of said normalization facial image information,

calculating a plurality of synthesized facial expression components corresponding to a specified facial expression component from said average facial expression component difference and said facial expression component corresponding to said applied facial image information, according to a facial expression of said applied facial image information and a specified facial expression,

generating normalization synthesized facial image information according to said plurality of synthesized facial expression components, and

deforming said normalization synthesized facial image information so that arrangement of reference points thereof are identical to arrangement of reference points for original said applied facial image information to generate synthesized facial image information.

8. The facial image information conversion method according to claim 7, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by Karhunen-Loeve Transform.

9. The facial image information conversion method according to claim 7, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by discrete cosine transform.

10. An image information conversion apparatus comprising:

means for extracting, from facial image information of a plurality of different facial expressions of a predetermined human being, a plurality of one-dimensional facial expression components for each said facial expression corresponding to respective projective components for a plurality of coordinate axes on facial image representation space with the entire face taken as a processing unit,

means for calculating a plurality of synthesized facial expression components corresponding to facial image information of a synthesized facial expression at an arbitrary rate for specified arbitrary two of said different facial expressions by interpolation/extrapolation of a facial expression component corresponding to said two different facial expressions, and

generating synthesized facial image information according to said plurality of synthesized facial expression components.

11. The facial image information conversion apparatus according to claim 10, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by Karhunen-Loeve Transform.

12. The facial image information conversion apparatus according to claim 10, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by discrete cosine transform.

13. A facial image information conversion apparatus comprising:

storage means for storing an average facial expression component difference between each facial expression provided by an average operation for a plurality of human beings according to facial expression components extracted from facial image information of a plurality of different facial expressions,

means for deforming arbitrary applied facial image information so that a plurality of reference points establish a predetermined position relationship with the entire face taken as a processing unit,

means for extracting a plurality of one-dimensional facial expression components corresponding to respective projective components for a plurality of coordinate axes on facial image representation space of said normalization facial image information,

means for calculating a plurality of synthesized facial expression components corresponding to a specified facial expression representation rate from an average facial expression component difference between each facial expression and said facial expression component corresponding to said applied facial image information according to a facial expression of said applied facial image information and a specified facial expression,

means for generating normalization synthesized facial image information according to said plurality of synthesized facial expression components, and

deforming said normalization synthesized facial image information so that arrangement of reference points of said normalization synthesized facial image information is identical to the arrangement of reference points of original said applied facial image information.

14. The facial image information conversion apparatus according to claim 13, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by Karhunen-Loeve Transform.

15. The facial image information conversion apparatus according to claim 13, wherein said facial expression components and said synthesized facial expression components are expansion coefficients of corresponding facial image information expanded by discrete cosine transform.