Face feature analysis for automatic lipreading and character animation

• US 6,028,960 A
• Filed: 09/20/1996
• Issued: 02/22/2000
• Est. Priority Date: 09/20/1996
• Status: Expired due to Term

• Current Assignee: Alcatel-Lucent USA, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Petajan, Eric David, Graf, Hans Peter
• Primary Examiner(s): Tran, Phuoc
• Application Number:
  US08/716,959
• Time in Patent Office: 1,250 Days
• Field of Search: 382/100, 382/103, 382/114, 382/118, 382/165, 382/172, 382/174, 382/203, 382/209, 382/253, 382/270
• US Class Current: 382/203
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A face feature analysis which begins by generating multiple face feature candidates, e.g., eyes and nose positions, using an isolated frame face analysis. Then, a nostril tracking window is defined around a nose candidate and tests are applied to the pixels therein based on percentages of skin color area pixels and nostril area pixels to determine whether the nose candidate represents an actual nose. Once actual nostrils are identified, size, separation and contiguity of the actual nostrils is determined by projecting the nostril pixels within the nostril tracking window. A mouth window is defined around the mouth region and mouth detail analysis is then applied to the pixels within the mouth window to identify inner mouth and teeth pixels and therefrom generate an inner mouth contour. The nostril position and inner mouth contour are used to generate a synthetic model head. A direct comparison is made between the inner mouth contour generated and that of a synthetic model head and the synthetic model head is adjusted accordingly. Vector quantization algorithms may be used to develop a codebook of face model parameters to improve processing efficiency. The face feature analysis is suitable regardless of noise, illumination variations, head tilt, scale variations and nostril shape.

Claims

1. A face feature analysis method for distinguishing features of a face of a subject in variable lighting conditions, comprising the steps of:

a) generating a plurality of eye, nose and mouth candidates within a current image frame which includes the face of the subject;

b) defining a nostril tracking window within the current image frame around one of the nose candidates, wherein said nostril tracking window is defined so as to exclude the eye and mouth candidates and to assure inclusion of nose nostrils of the subject in a next frame after a fastest possible head movement between said current and next frames; and

c) analyzing each pixel within said nostril tracking window to determine whether the nose candidate enclosed within said nostril tracking window represents actual nostrils by performing a skin color area test and a nostril area test.

2. The method of claim 1, wherein said skin color area test includes the steps of:

(d) comparing an intensity of each pixel within said nostril tracking window to a range of pixel intensities corresponding to a range of possible skin colors of the face of the subject and classifying the pixel as a skin color area pixel if the intensity is within the range of possible skin colors;

(e) identifying a percentage area of skin color area pixels within the nostril tracking window; and

(f) comparing said percentage area of skin color area pixels to an acceptable range of percentage area of skin color area pixels, wherein if said percentage area of skin color area pixels is within the acceptable range of percentage area of skin color area pixels then actual nostrils have been detected within said nostril tracking window, and if said percentage area of skin color area pixels is outside the acceptable range of percentage area of skin color area pixels then the actual nostrils are not enclosed within said nostril tracking window and said step c) is repeated to define a nostril tracking window around another nose candidate and steps d) through f) are repeated after said repeated step c).

3. The method of claim 1, wherein the nostril area test includes the steps of:

d) binarizing each pixel within said nostril tracking window by comparing the intensity of each pixel within said nostril tracking window to one color nostril threshold in a range of color nostril thresholds and classifying the pixel as a nostril pixel if the intensity is greater than the color nostril threshold;

c) identifying an area percentage of nostril pixels within said nostril tracking window;

f) comparing said area percentage of nostril pixels to an acceptable range of area percentage of nostril pixels, wherein if said area percentage of nostril pixels is outside the acceptable range of area percentage of nostril pixels then the color nostril threshold is adjusted, and if the adjusted color nostril threshold value is within the range of color nostril threshold values then steps d) through f) are repeated, and if the adjusted nostril threshold value is outside the range of nostril threshold values then the actual nostrils have not been located within said nostril tracking window and said nostril tracking window is defined around another nose candidate; and

wherein, if said area percentage of nostril pixels is within the acceptable range of area percentage of nostril pixels then the actual nostrils are verified as being enclosed within said nostril tracking window and said nostril threshold value represents a final selected nostril threshold value.

4. A face feature analysis method for distinguishing features of a face of a subject in variable lighting conditions, comprising the steps of:

a) generating a plurality of eye, nose and mouth candidates within a current image frame which includes the face of the subject;

b) defining a nostril tracking window within the current image frame around one of the nose candidates, wherein said nostril tracking window is defined so as to exclude the eye and mouth candidates and to assure inclusion of nose nostrils of the subject in a next frame after a fastest possible head movement between said current and next frames; and

c) analyzing each pixel within said nostril tracking window to determine whether the nose candidate enclosed within said nostril tracking window represents actual nostrils by;

d) comparing an intensity of each pixel within said nostril tracking window to a range of pixel intensities corresponding to a range of possible skin colors of the face of the subject and classifying the pixel as a skin color area pixel if the intensity is within the range of possible skin colors;

e) identifying a percentage area of skin color area pixels within the nostril tracking window;

f) comparing said percentage area of skin color area pixels to an acceptable range of percentage area of skin color area pixels, wherein if said percentage area of skin color area pixels is within the acceptable range of percentage area of skin color area pixels then actual nostrils have been detected within said nostril tracking window, and if said percentage area of skin color area pixels is outside the acceptable range of percentage area of skin color area pixels then the actual nostrils are not enclosed within said nostril tracking window and said step c) is repeated to define a nostril tracking window around another nose candidate and steps d) through f) are repeated after said repeated step c).

5. The face feature analysis in claim 4, wherein step c) further comprises the step of verifying that the actual nostrils have been detected within said nostril tracking window, said verification step further comprising the steps of;

g) binarizing each pixel within said nostril tracking window by comparing the intensity of each pixel within said nostril tracking window to one color nostril threshold in a range of color nostril thresholds and classifying the pixel as a nostril pixel if the intensity is greater than the color nostril threshold;

h) identifying an area percentage of nostril pixels within said nostril tracking window;

i) comparing said area percentage of nostril pixels to an acceptable range of area percentage of nostril pixels, wherein if said area percentage of nostril pixels is outside the acceptable range of area percentage of nostril pixels, then the color nostril threshold is adjusted and if the adjusted color nostril threshold value is within the range of color nostril threshold values then steps g) through i) are repeated whereas, if the adjusted nostril threshold value is outside the range of nostril threshold values then the actual nostrils have not been located within said nostril tracking window and said nostril tracking window is defined around another nose candidate; and

wherein, if said area percentage of nostril pixels is within the acceptable range of area percentage of nostril pixels then the actual nostrils are verified as being enclosed within said nostril tracking window and said nostril threshold value represents a final selected nostril threshold value.

6. The face feature analysis method in claim 5, further comprising the step of determining nostril values of the actual nostrils enclosed within said nostril tracking window and matching said nostril values to a template, said step of determining said nostril values comprising the steps of:

j) projecting in a horizontal and in a vertical direction each pixel within said nostril tracking window for each horizontal and vertical projection threshold in a range of horizontal and vertical projection thresholds to generate at least one horizontal projection run and one vertical projection run;

k) generating said nostril size values for each horizontal and vertical projection threshold from a run length of projection values which exceed said vertical and horizontal projection thresholds;

l) comparing said nostril size values for each horizontal and vertical projection threshold in the range of horizontal and vertical projection thresholds to nostril size values of each template in a set of possible template values to find a best match template; and

m) scaling the best match template based on nostril separation of a previous frame.

7. The face feature analysis method in claim 6, wherein said step l) further comprises the steps of:

n) assigning a score to each comparison of said nostril size values for each horizontal and vertical projection threshold to the nostril size values for each template, wherein said score is defined as

space="preserve" listing-type="equation">score=(0.1×

(abs(left nostril width-template left nostril width)+abs(right nostril width-template right nostril width)))+(0.2×

(abs(right nostril centerline-left nostril

space="preserve" listing-type="equation">centerline-template nostril separation)))+(0.2×

(abs(nostril height-template nostril height)))+(0.5×

(abs((nostril width/nostril height)-8.0)))+(0.5×

(abs(((right nostril center line-left nostril centerline)/nostril height)-2.0)));

o) storing in a memory a current score based on one template;

p) comparing the current score with another next score generated using another template and updating said memory with a better score of the current and next scores, and repeating steps n) through p) for each horizontal and vertical projection threshold in the range of horizontal and vertical projection thresholds and for each template value within the range of template values.

8. The face feature analysis method in claim 6, further comprising the step of orienting said nostril tracking window about the actual nostrils detected therein, said orienting step comprising the steps of:

q) centering said nostril tracking window about the actual nostrils; and

r) rotating said nostril tracking window relative to the actual nostrils to compensate for head tilt of the subject.

9. The face feature analysis method in claim 8, further comprising the step of generating a closed mouth line, said generating step comprising the steps of:

s) defining a mouth window in the image frame relative to said centered and rotated nostril tracking window;

t) converting each color pixel within said mouth window to a monochrome pixel by averaging R, G and B color components of each pixel;

u) finding a minimum luminance as a function of horizontal position for each column of pixels within said mouth window to locate a mouth line;

v) generating a modified skin color range by adjusting the skin color range based on the final selected nostril threshold value to further distinguish skin color from shadows;

w) defining a closed mouth window within said mouth window to include pixels bounded vertically between centers of each of the actual nostrils and bounded horizontally in a range seven pixels wide, said range representing three pixels above and three pixels below the mouth line and including the mouth line;

x) comparing an intensity of each pixel within said closed mouth window to said modified skin color range and classifying the pixel as a modified skin color pixel if the intensity is within said modified skin color range;

y) finding an area percentage of modified skin color pixels within said closed mouth window;

z) comparing said area percentage of modified skin color pixels to an acceptable range of area percentage of modified skin color pixels and, if said area percentage of modified skin color pixels is within the acceptable range of area percentage of modified skin color pixels, then a closed mouth position has been detected; and

aa) defining a closed mouth line based on the pixels in said closed mouth window which are non-modified skin color area pixels.

10. The face feature analysis method in claim 9, further comprising the step of identifying inner mouth pixels within said mouth window, said mouth pixels identifying step comprising the steps of:

bb) setting an inner lip color threshold array to a range of minimum color intensity values as a function of horizontal position for each column within said mouth window;

cc) comparing for each said column an intensity of each pixel within said mouth window to the respective inner lip color threshold for the said each column of pixels and classifying as an inner mouth pixel if the intensity is less than said inner lip color threshold; and

dd) analyzing each inner mouth pixel within said mouth window to determine if the inner mouth pixel is an isolated inner mouth pixel and ignoring isolated inner mouth pixels so that only the inner mouth pixels are detected.

11. The face feature analysis method in claim 10, further comprising the step of identifying teeth pixels within said mouth window, said teeth pixels identifying step comprising the steps of:

ee) defining an inner mouth window within the image frame and enclosing said inner mouth pixels;

ff) defining a teeth search window within said inner mouth window, said teeth search window corresponding in height to the height of said inner mouth window and bounded horizontally between centers of the actual nostrils; and

gg) analyzing each pixel within said inner mouth window to determine whether it is a teeth pixel, said analyzing step comprising the step of comparing an intensity of each pixel within said teeth search window to pixel intensities corresponding to a teeth color range and calculating a difference between each R, G and B color component for each pixel within said teeth search window, wherein said each pixel is classified as a teeth pixel if the intensity is within the teeth color range and the difference between each color component is less than or equal to 30.

12. The face feature analysis method in claim 11, further comprising the step of generating an inner lip contour, said inner lip contour generating step comprising the step of:

hh) generating an upper inner lip contour geometrically represented by curves beginning at each mouth corner and having at any given moment one of increasing and constant slope towards a facial centerline so that said upper inner lip contour encloses said inner mouth and teeth pixels; and

ii) generating a lower inner lip contour geometrically represented by a curve having at any given moment one of increasing and constant slope relative to the curves representing said upper inner lip contour so that said lower inner lip contour encloses said inner mouth and teeth pixels.

13. The face feature analysis method in claim 12, further comprising the step of mapping the inner lip contour to a generated synthetic head and adjusting the synthetic head to correspond to the inner lip contour, said step of mapping comprising the steps of:

jj) generating a synthetic model head based on said nostril size values and said inner lip contour;

kk) directing comparing said inner lip contour with an inner lip contour of said synthetic model head by finding a distance measure representing a difference between a first vertical distance between said upper and lower inner lip contours and a second vertical distance between said upper and lower inner lip contours of said synthetic model head; and

ll) adjusting the inner lip contour of said synthetic model head and repeating step jj) until a best match is achieved.

14. The face feature analysis method in claim 13, wherein said steps kk) and ll) are performed once for a set of samples of every mouth position to generate a plurality of codebook vectors of synthetic model mouth parameters using a vector quantization clustering algorithm and then directly comparing the upper and lower inner lip contours with synthetic upper and lower inner lip contours stored within each codebook vector.

15. The face feature analysis method in claim 6, further comprising the steps, when two vertical and horizontal projection runs are generated, of:

mm) defining a horizontal line bisecting each horizontal projection run and a vertical line bisecting each vertical projection run to thereby generate a four intersections within said nostril tracking window, wherein each of the four intersections represents a possible location of one of the actual nostrils;

nn) binarizing pixels surrounding each of the four intersections based on said final selected nostril threshold value and classifying the pixels as nostril pixels if the pixel has an intensity greater than said final selected nostril threshold value; and

oo) classifying two of the four intersections as representing the actual nostrils in which the nostril pixels are detected, wherein the two intersections in which the actual nostrils are detected are diagonally opposed to one another.

16. A face feature analysis method for distinguishing features of a face of a subject in variable lighting conditions, comprising the steps of:

a) generating an actual inner lip contour including an actual upper and an actual lower inner lip contour;

b) generating a synthetic inner lip contour including a synthetic upper and a synthetic lower inner lip contour based on said actual upper and lower inner lip contours;

c) directly comparing said actual inner lip contour with said synthetic inner lip contour by determining a distance measure representing a difference between a vertical distance between said actual upper and lower inner lip contours and a vertical distance between said synthetic upper and lower inner lip contour; and

d) adjusting said synthetic inner lip contour and repeating step c) until a best match is achieved.

17. The method for face feature analysis in claim 16, wherein step c) is performed once for a set of samples of every mouth position to generate a plurality of codebook vectors of synthetic model mouth parameters using a vector quantization clustering algorithm and then directly comparing said upper and lower inner lip contours to the synthetic model mouth parameters stored within each codebook vector.