Method and apparatus for processing images of facial expressions

• US 5,689,575 A
• Filed: 11/21/1994
• Issued: 11/18/1997
• Est. Priority Date: 11/22/1993
• Status: Expired due to Term

• Current Assignee: Rakuten, Inc. (Rakuten Group, Inc.)
• Original Assignee: Hitachi, Ltd.
• Inventors: Sako, Hiroshi, Whitehouse, Mark, Smith, Anthony
• Primary Examiner(s): BOST, DWAYNE D
• Assistant Examiner(s): Johnson, Brian L.
• Application Number:
  US08/346,164
• Time in Patent Office: 1,093 Days
• Field of Search: 382/115, 382/118, 382/164, 382/165, 382/170, 382/171, 382/173, 382/190, 382/191, 382/303
• US Class Current: 382/118
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A real time output containing data relating to states of facial parts is generated. A facial area detection unit (11) has monitoring (51-57) and determining (59-61) processing circuits operating in a pipelining manner to determine position of the facial area. The monitoring circuits (51-57) monitor pixel value frequency using 3D histogram and backprojection processing. The generating facial area signal has masks applied by a unit (12) which supplies data to mouth area and eye area detection units (14,15). Each of these operate on similar principles to the facial area detection unit (11).

Claims

1. An image processing apparatus comprising:

means for receiving a color facial image data stream in digital format;

a plurality of image data processing circuits interconnected for real time processing the image data in a pipelining manner to provide a real time output, the circuits comprising;

means for monitoring color values in the image data to identify image data pixels representing facial parts, and monitoring means comprising means for carrying out three-dimensional color histogram matching to monitor frequency of occurrence of pixel values in the image data and a backprojection means for comparing a generated histogram with a template histogram generated off-line, and

means for determining states of the facial parts by monitoring positional coordinates of the identified pixels, said means for determining further comprising a counter for determining the area of the facial part or parts; and

an output device outputting the determined facial part state data in real time.

2. An apparatus as claimed in claim 1 further comprising means for normalizing the received color facial image data stream.

3. An apparatus as claimed in claim 1 wherein the processing circuits comprise:

a facial area detection unit comprising circuits for carrying out facial area detection operations on the image data; and

a facial part detection unit connected to said facial area detection unit and comprising processing circuits for determining states of a facial part within the detected facial area using the image data and an output signal from said facial area detection unit.

4. An apparatus as claimed in claim 3 further comprising a mask generating circuit connected between the facial area and facial part detection units.

5. An apparatus as claimed in claim 3 comprising two facial detection units, including:

a mouth area detection unit; and

an eye area detection unit.

6. An apparatus as claimed in claim 5 wherein the apparatus further comprises a mask generating circuit connected between the facial area detection unit and the facial part detection units, said mask generating circuit unit comprising means for generating an eye mask signal and a mouth mask signal.

7. An apparatus as claimed in claim 5 wherein the mouth area and eye area detection units are connected in the apparatus to operate on the same image data in parallel.

8. An apparatus as claimed in claim 1, wherein the apparatus further comprises means for carrying out a consistency check to validate determined positional data.

9. An apparatus as claimed in claim 8 wherein each processing circuit comprises means for carrying out a consistency check on positional data which it generates.

10. An apparatus as claimed in claim 8 wherein the consistency check means comprises a processor programmed to compare the positional data with reference positional data.

11. An apparatus as claimed in claim 1 wherein the monitoring means comprises means for carrying out binary erosion and binary dilation steps to generate image data in which noise is reduced and the subject area is recovered.

12. An apparatus as claimed in claim 1 further comprising image data processing circuits for determining location of a colored fingertip represented in the input image data.

13. An apparatus as claimed in claim 1 comprising image processing circuits for facial area and separate image processing circuits for colored fingertips, said circuits being connected to process the input image data in parallel.

14. An apparatus as claimed in claim 12 further comprising a result processor comprising means for receiving facial part and colored fingertip positional data to generate an output signal representing proximity of a colored fingertip to facial parts for assistance in communication of sign language.

15. An apparatus as claimed in claim 1 further comprising a video device for generating the image data stream.

16. An apparatus as claimed in claim 15, wherein the video device is a camera having an analog to digital converter.

17. An apparatus as claimed in claim 15 wherein the video device is a camera, and the camera is mounted on a frame adjacent a light source so that it is directed upwardly at an angle so that the face of a person seated next to the frame is within the field of view.

18. A method of processing facial image data, the method comprising the steps of:

receiving a digital color image data stream;

monitoring color values in the image data to identify image data pixels representing facial parts, said monitoring step comprising sub-steps of carrying out three-dimensional color histogram matching in which there is backprojection with comparison of a generated three-dimensional color histogram with a template histogram generated off-line;

determining states of the facial parts by monitoring positional coordinates of the identified pixels and using a counter to determine the area of the facial parts, said monitoring and determining steps being carried out by processing circuits interconnected for real time processing the image data in a pipelining manner; and

outputting in real time a signal representing the determined facial part state data.

19. A method as claimed in claim 18, wherein the monitoring and determining steps are carried out to initially identify facial area and position, and subsequently to identify facial parts and determine states of the facial parts.