Facial image processing

• US 5,870,138 A
• Filed: 03/14/1996
• Issued: 02/09/1999
• Est. Priority Date: 03/31/1995
• Status: Expired due to Term

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Abe, Masahiro, Smith, Anthony, Sako, Hiroshi, Sutherland, Alistair
• Primary Examiner(s): Garber, Wendy
• Assistant Examiner(s): ONUAKU, CHRISTOPHER O
• Application Number:
  US08/615,360
• Time in Patent Office: 1,062 Days
• Field of Search: 348/17, 348/152, 348/169, 348/578, 382/118, 382/100, 382/254, 382/286, 382/276, 382/103, 345/435
• US Class Current: 348/143
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

A camera (10) captures a subject facial image. A unit (30) generates a tracking signal containing feature extraction data. In conjunction with a memory store (40), the unit (30) generates an output signal representing a substitute face having the features of the subject face. To generate the tracking signal, the facial area is detected (3103, 3107-8) initially and used in location of the mouth (3102, 3110-3114) and eyes (3104-6, 3115-3118). Only the H and S components are used for detecting the facial area; and only the S and V components for the mouth within the facial area. A face vector may be generated (50) using the tracking signals.

Claims

1. An image processing method comprising the steps of receiving a subject facial image signal, generating a feature extraction tracking signal, and processing said tracking signal to provide a processed output signal, wherein:

said processing step comprises the sub-steps of;

generating an image signal representing a substitute face; and

modifying said substitute face image signal in real time according to the tracking signal to generate an output signal representing the substitute face with facial features of the subject face by converting the tracking signal to a facial characteristic signal representing positional characteristics of facial features, the facial characteristic signal being a vector signal wherein pixel change data in an update image memory is selected by comparison of current and previous characteristics signals.

3. A method as claimed in claim 1 wherein th facial characteristic signal is a vector signal.

4. A method as claimed in claim 1, wherein the facial characteristic signal is a vector signal and wherein the previous characteristic signal is stored in a dedicated register.

6. A method as claimed in claim 1, wherein the step of generating the substitute face image signal comprises the steps of retrieving from an image memory a primeval image and subsequently updating said image according to the tracking signal.

7. A method as claimed in claim 6, wherein the primeval image is transferred from a primeval image memory to the output image memory upon initialization.

8. A method as claimed in claim 1, wherein the tracking signal is generated by:

generating a subject facial image signal in H,S,V format;

passing at least two components of said H,S,V format signal through a band pass filter;

mapping the filter output signals over the subject facial image pixel area; and

determining feature locations within the pixel area according to the mapping of the filter output signals.

9. A method as claimed in claim 8, wherein the step of determining feature locations comprises the sub-steps of detecting the facial area and subsequently detecting mouth and eye areas within the detected facial area.

10. A method as claimed in claim 8, wherein the step of determining feature locations comprises the sub-steps of detecting the facial area and subsequently detecting mouth and eye areas within the detected facial area, and wherein only the H and S components of the subject facial image signal are processed for facial area detection.

11. A method as claimed in claim 12, wherein the step of determining feature locations comprises the sub-steps of detecting the facial area and subsequently detecting mouth and eye areas within the detected facial area, and wherein only the H and S components of the subject facial image signal are processed for facial area detection, and said processing comprises band pass filtering followed by mapping the filter output over the pixel area.

12. A method as claimed in claim 8, wherein the step of determining feature locations comprises the sub-steps of detecting the facial area and subsequently detecting mouth and eye areas within the detected facial area, and mouth detection is achieved by processing the S and V components of the subject image signal within the detected facial area.

13. A method as claimed in claim 8, wherein the step of determining feature locations comprises the sub-steps of detecting the facial area and subsequently detecting mouth and eye areas within the detected facial area, and wherein only the V component of the subject image signal is processed for eye detection.

14. A method as claimed in claim 8, wherein the step of determining feature locations comprises the sub-steps of detecting the facial area and subsequently detecting mouth and eye areas within the detected facial area, and only the V component of the subject image signal is processed for eye detection, and wherein the V component is normalized and eye detection is achieved by correlation.

2. A method as claimed in 1 wherein the output signal is transmitted from an output image memory which is modified by overwriting pixel values transferred from an update image memory in response to the tracking signal, the update image memory storing pixel changes between frames the update transfers being controlled by reference to a lookup table having pointers to addresses in the update image memory.

5. A method as claimed in claim 2, wherein the update image memory stores data elements in three fields, namely an indicator of the position of the element, a second field having an address for a location in the output image memory and a third field having a color value for the pixel.

15. An image processing method comprising the steps of:

receiving a subject facial image signal;

extracting features in the signal to generate a tracking signal;

transforming the tracking signal to a facial characteristic signal representing positional characteristics of features of the subject face;

writing a primeval substitute image to an output image memory; and

modifying the output image memory by overwriting pixel values transferred from an update image memory, the facial characteristic signal being a vector signal wherein pixel change data in said output image memory is selected by comparison of current and previous characteristics signals, said transfers being controlled in response to the facial characteristic signal.

16. An image processing apparatus comprising means for receiving a subject facial image signal, means for generating a feature extraction tracking signal, and means for processing said tracking signal to provide a processed output signal, said processing means comprising means for:

generating an image signal representing a substitute face; and

modifying said substitute face image signal in real time according to the tracking signal to generate an output signal representing the substitute face with facial features of the subject face by converting the tracking signal to a facial characteristic signal representing positional characteristics of facial features, the facial characteristic signal being a vector signal wherein pixel change data in an update image memory is selected by comparison of current and previous characteristic signals.

17. An image processing apparatus as claimed in claim 16, wherein said processing means comprises an output image memory, an update image memory and means for modifying said output image memory by transferring pixel values from the update image memory in response to the tracking signal.

18. An image processing apparatus as claimed in claim 17, wherein the update image memory stores pixel changes between frames.

19. An image processing apparatus as claimed in claim 16, wherein said processing means comprises means for converting the tracking signal to a facial characteristic signal representing positional characteristics of facial features.

20. An image processing apparatus as claimed in claim 19, wherein said facial characteristic signal is a vector signal.

21. An image processing apparatus as claimed in claim 16, wherein the processing means further comprises a neural network device for training during operation of the apparatus by monitoring feature data and for subsequently providing feedback to assist in recognition of features.

22. An image processing apparatus comprising:

means for receiving a subject facial image signal;

means for extracting features in the signal to generate a tracking signal; and

processing means comprising;

means for transforming the tracking signal to a facial characteristic signal representing positional characteristics of features of the subject face;

an output image memory;

means for writing a primeval substitute image to the output image memory;

an update image memory; and

means for modifying the output image memory by transferring pixel values from the update image memory to the output image memory in response to the facial characteristic signal.

23. An image processing method comprising the steps of receiving an input image signal and generating a feature extraction tracking signal, characterised in that:

the input image signal is in H,S,V format;

a facial area location signal is generated by passing at least part of the input image signal through a band pass filter and analyzing the output of the filter;

a mouth location signal is generated by passing at least part of the input image signal through a band pass filter and analyzing the output of the filter within the facial pixel area according to the facial area location signal;

eye location signals are generated by processing at least part of the input image signal within the facial pixel area according to the facial area location signal; and

the facial area location, mouth location and eye location signals are outputted as output tracking signals.

24. A method as claimed in claim 23, wherein only two of the H,S,V input image components are used for generation of the facial area location signal.

25. A method as claimed in claim 23, wherein the H and S components are passed through the band pass filter for generation of the facial area location signal.

26. A method as claimed in claim 23 wherein only two of the H,S,V input image components are used for generation of the mouth location signal.

27. A method as claimed in claim 26, wherein the S and V components are passed through the band pass filter for generation of the mouth area location signal.

28. A method as claimed in claim 23, wherein the band pass filter output signals are analyzed by mapping the output data over the pixel area and generating a projection in a mapping axis and analysing said projection.

29. A method as claimed in claim 28, wherein two projections are generated, one for each axis in a two-dimensional pixel area plane.

30. A method as claimed in claim 23, wherein each band pass filter comprises a look-up table containing filter indicators which are generated off-line.

31. A method as claimed in claim 28, wherein the step of analyzing the filter output signals comprises the further steps of determining maximum limits in the pixel area for a feature and generating a bounding box according to said limits.

32. A method as claimed in claim 23, wherein the image processing for generation of the eye area location signals comprises the steps of correlation with templates.

33. A method as claimed in claim 32, wherein the image signal is normalized before correlation.

34. A method as claimed in claim 32, wherein the V component only of the input image signal is used for generation of the eye location signals.

35. A method as claimed in claim 23, wherein the tracking signals which are generated are post-processed to generate a facial characteristic signal representing both location and positional characteristic data, said signal being generated by passing the tracking signals through logic devices.

36. An image processing apparatus comprising:

means for receiving an input image signal in H,S,V format;

a facial area band pass filter;

means for passing at least part of the input image signal through the facial area band pass filter and analyzing the output of the filter to generate a facial area location signal;

a mouth location band pass filter;

means for passing at least part of the input image signal through the mouth location band pass filter and for analyzing the output of the filter within the face pixel area according to the facial area location signal;

processing means for processing at least part of the input image signal within the facial pixel area according to the facial area location signal to generate eye location signals; and

means for outputting said facial area location, mouth location, and eye location signals as output tracking signals.

37. An apparatus as claimed in claim 36, wherein only the H and S components of the input image signal are passed through the facial area band pass filter.

38. An apparatus as claimed in claim 36, wherein only the S and V components of the input image signal are passed through the mouth location band pass filter.

39. An apparatus as claimed in claim 36, further comprising post-processing logic devices comprising means for receiving the tracking signals and generating a facial characteristic signal representing both location and positional characteristic data.