System and method for measuring and quantitating facial movements

• US 5,482,048 A
• Filed: 06/30/1993
• Issued: 01/09/1996
• Est. Priority Date: 06/30/1993
• Status: Expired due to Fees

• Current Assignee: University Of Pittsburgh (University of Pittsburgh of The Commonwealth System of Higher Education)
• Original Assignee: University Of Pittsburgh (University of Pittsburgh of The Commonwealth System of Higher Education)
• Inventors: Johnson, Peter C.
• Primary Examiner(s): Hayes, Gail O.
• Assistant Examiner(s): Shingala, Gita D.
• Application Number:
  US08/085,283
• Time in Patent Office: 923 Days
• Field of Search: 364/413.13, 364/413.02, 364/413.27, 382/6, 382/51, 128/665, 128/774, 128/782
• US Class Current: 600/476
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

A system and method for measuring and quantitating facial movements useful in detecting, characterizing and treating facial paralysis or abnormal facial movements. A preferred embodiment of the system comprises a plurality of markers applied to a plurality of facial zones, apparatus for recording images of the marked face at repose and for recording images of the marked face at selected maximal facial movements, apparatus for assigning location coordinates to the markers of each recorded facial image, apparatus for converting location coordinates of each marker of each facial image to actual positions on the face, apparatus for determining the change in actual marker position on the face at each selected maximal facial movement and apparatus for displaying the actual marker position changes for each selected facial movement. A preferred embodiment of the method comprises the steps of applying a plurality of markers to a plurality of facial zones, recording images of the marked face at repose and recording images of the marked face at selected maximal facial movements, assigning location coordinates to the markers of the facial images, converting the location coordinates of each marker to actual positions on the face, determining the change in actual marker position for each selected maximal facial movement, and displaying the actual positional changes of each marker for each selected facial movement.

Claims

1. A system for measuring and quantitating movements of a face, the system comprising:

at least one marker applied to at least one specific zone of the face to track motion of the specific zone of the face;

apparatus for recording an image of the marked face at repose and for recording an image of the marked face at at least one selected facial position, said selected position being the result of facial movement;

apparatus for assigning at least one location coordinate to the marker of the facial image at repose and for assigning at least one location coordinate to the marker of the facial image at the selected facial position;

means for converting the location coordinate of the marker to actual positions on the face;

means for determining a change in actual marker position; and

means for displaying the change in actual marker position.

2. The system of claim 1 wherein the marker includes an adhesive dot.

3. The system of claim 1 wherein the apparatus for recording facial images is a camera which captures the facial images on film developed into photographs.

4. The system of claim 1 wherein the apparatus for recording facial images is a camera which captures the facial images on film developed into slides;

and

the system further comprises apparatus for viewing the slides.

5. The system of claim 1 wherein the apparatus for recording facial images is a video camera which captures the facial images on videotape;

and

the system further comprises apparatus for viewing the videotape.

7. The system of claim 1 wherein the apparatus for assigning location coordinates to the marker in the facial images includes

a grid in juxtaposition with the facial images whereby the location coordinate is assigned to the marker of each facial image based on the grid orientation.

8. The system of claim 1 wherein the apparatus for assigning location coordinates to the marker in the facial images includes

a digitizer computer program;

a data input device; and

a digitizer board;

whereby the data input device is selectively aligned with the marker of each facial image and communicates with the digitizer board and the digitizer computer program to assign the location coordinates to the marker.

9. The system of claim 1 wherein the means for converting the location coordinates of the marker of each facial image to actual positions on the face includes a computer spreadsheet.

10. The system of claim 1 wherein the means for determining a change in actual marker position includes a computer spreadsheet.

11. The system of claim 1 wherein the means for displaying the change in actual marker position includes a graph.

12. The system of claim 1 further comprising

means positioned pro ate to the face for calibrating the facial images.

13. The system of claim 1 further comprising

a calibration ruler positioned proximate to a tip of a nose of the face by positioning means;

and

a reference marker applied to the nose behind and partially above the calibration ruler such that the reference marker partially appears above the calibration ruler when the face is viewed head-on.

6. The system of claim rein the apparatus for recording facial images is a compact disc camera which captures the facial images on digitally based media;

and

the system further comprises apparatus for viewing the digitally based media.

14. A method for measuring and quantitating movements of a face, the method comprising the step of:

applying at least one marker to at least one specific zone of the face to track motion of the specific zone of the face;

recording an image of the marked face at repose and recording an image of the marked face at at least one selected facial position, said selected position being the result of facial movement;

assigning at least one location coordinate to the marker of the facial image at repose and assigning at least one location coordinate to the marker of the facial image at the selected facial position;

converting the location coordinates of the marker to actual positions of the face;

determining a change in actual marker position on the face; and

displaying the change in actual marker position on the face.

15. The method of claim 14 further comprising the step of:

centering pupils of the face prior to applying a marker to facial zones including radix, philtrum, mentum, left supraorbital, right supraorbital, left infraorbital, right infraorbital, left oral commissure and right oral commissure zones.

16. The method of claim 14 further comprising the steps of:

prior to recording the facial image at repose,

positioning a calibration ruler proximate to a tip of a nose of the face by positioning means;

applying a reference marker to the nose behind and partially above the calibration ruler such that the reference marker partially appears above the calibration ruler when the face is viewed head-on; and

focusing on the calibration ruler when recording the facial images.

17. The method of claim 14 wherein the selected facial movement includes maximal brow lift, maximal tight eye closure, maximal smile, maximal frown and maximal mouth pucker.

18. The method of claim 14 herein the step of assigning location coordinates to the marker of each facial image includes the steps of:

juxtapositioning a grid to each facial image so that a horizontal axis of the grid passes through lower limbs of a medial canthi of the facial image and a vertical axis of the grid passes through a perceived center of the bony nasal dorsum of the facial image; and

recording location coordinates for each marker of each facial image based on the grid orientation.

19. The method of claim 14 wherein the step of assigning location coordinates to the marker of each facial image includes the steps of:

projecting each facial image onto a digitizer board; and

selectively aligning a data input device with the marker of each facial image, the data input device communicating with the digitizer board and a digitizer computer program to assign location coordinates to the marker of each facial image.

20. The method of claim 14 wherein the step of assigning location coordinates to the marker of each facial image includes the steps of:

projecting each facial image onto a computer screen; and

selectively aligning a data input device with the marker of each facial image, the data input device communicating with the computer screen and a digitizer computer program to assign location coordinates to the marker of each facial image.

21. The method of claim 14 wherein the step of converting the location coordinates of the marker of each facial image to actual positions on the face includes the step of:

inputting the location coordinates of the marker of each facial image into a computer spreadsheet adapted to calculate the actual position of the marker of each facial image.

22. The method of claim 14 wherein the step of determining a change in actual marker position on the face includes the step of:

inputting the actual marker positions of each facial image into a computer spreadsheet adapted to calculated the change in actual marker position.

23. The method of claim 14 wherein the step of displaying the change in actual marker position on the face includes the step of:

graphing the change in actual marker position.

24. A method for measuring and quantitating movements of a face, the method comprising the steps of:

centering pupils of the face at repose;

applying a marker to a plurality of facial zones including radix, philtrum, mentum, left supraorbital, right supraorbital, left infraorbital, right infraorbital, left oral commissure and right oral commissure zones;

positioning a calibration ruler proximate to a tip of a nose of the face by positioning means, said calibration ruler including at least one calibration unit;

applying a reference marker to the nose behind and partially above the calibration ruler so that the reference marker partially appears above the calibration ruler when the face is viewed head-on;

recording an image of the marked face at repose focused on the calibration ruler using a camera capable of capturing facial images on film for developing slides;

using said camera and focusing on said calibration ruler, recording an image of the marked face at a plurality of selected facial movements including maximal brow lift, maximal tight eye closure, maximal smile, maximal frown and maximal mouth pucker;

developing said facial images into slides;

viewing each facial image slide through a slide viewer;

juxtapositioning a grid to each facial image displayed by the slide viewer so that a horizontal axis of the grid passes through lower limbs of a medial canthi of the facial image and a vertical axis of the grid passes through a perceived center of a bony nasal dorsum of the facial image, said grid including at least one grid unit;

recording location coordinates for each marker of each facial image based on the grid orientation;

calibrating each facial image by placing the grid proximate to the image of the calibration ruler of each facial image so as to determine the number of grid units per calibration unit for that facial image;

converting the location coordinates of each marker to actual positions on the face relative to a trans-canthal/trans-nasal origin formed by the medial canthi and the bony nasal dorsum of the facial image;

determining a change in actual marker position for each marker on the face for each selected facial movement; and

graphing the change in actual marker position on the face for each selected facial movement.

25. A method for measuring and quantitating movements of a face, the method comprising the steps of:

centering pupils of the face at repose;

applying a marker to a plurality of facial zones including radix, philtrum, mentum, left supraorbital, right supraorbital, left infraorbital, right infraorbital, left oral commissure and right oral commissure zones;

positioning a calibration ruler proximate to a tip of a nose of the face by positioning means, said calibration ruler including at least one calibration unit;

applying a reference marker to the nose behind and partially above the calibration ruler so that the reference marker partially appears above the calibration ruler when the face is viewed head-on;

recording an image of the marked face at repose focused on the calibration ruler using a camera capable of capturing facial images on film for developing photographs;

using said camera and focusing on said calibration ruler, recording an image of the marked face at a plurality of selected facial movements including maximal brow lift, maximal tight eye closure, maximal smile, maximal frown and maximal mouth pucker;

developing said facial images into photographs;

juxtapositioning a grid to each photograph so that a horizontal axis of the grid passes through lower limbs of a medial canthi of the facial image and a vertical axis of the grid passes through a perceived center of the bony nasal dorsum of the facial image, said grid including at least one grid unit;

recording location coordinates for each marker of each facial image based on the grid orientation;

calibrating each facial image by placing the grid proximate to the image of the calibration ruler of each facial image so as to determine the number of grid units per calibration unit for that facial image;

converting the location coordinates of each marker to actual positions on the face relative to a trans-canthal/trans-nasal origin formed by the medial canthi and the bony nasal dorsum of the facial image;

determining a change in actual marker position for each marker on the face for each selected facial movement; and

graphing the change in actual marker position on the face for each selected facial movement.

26. A method for measuring and quantitating movements of a face, the method comprising the steps of:

centering pupils of the face at repose;

applying a marker to a plurality of facial zones including radix, philtrum, mentum, left supraorbital, right supraorbital, left infraorbital, right infraorbital, left oral commissure and right oral commissure zones;

positioning a calibration ruler proximate to a tip of a nose of the face by positioning means, said calibration ruler including at least one calibration unit;

applying a reference marker to the nose behind and partially above the calibration ruler so that the reference marker partially appears above the calibration ruler when the face is viewed head-on;

recording an image of the marked face at repose focused on the calibration ruler using a video camera capable of capturing facial images on videotape;

using said video camera and focusing on said calibration ruler, recording an image of the marked face at a plurality of selected facial movements including maximal brow lift, maximal tight eye closure, maximal smile, maximal frown and maximal mouth pucker;

viewing each facial image by using a videotape player;

juxtapositioning a grid to each facial image displayed by the videotape player so that a horizontal axis of the grid passes through lower limbs of a medial canthi of the facial image and a vertical axis of the grid passes through a perceived center of the bony nasal dorsum of the facial image, said grid including at least one grid unit;

recording location coordinates for each marker of each facial image based on the grid orientation;

calibrating each facial image by placing the grid proximate to the image of the calibration ruler of each facial image so as to determine the number of grid units per calibration unit for that facial image;

converting the location coordinates of each marker to actual positions on the face relative to a trans-canthal/trans-nasal origin formed by the medial canthi and the bony nasal dorsum of the facial image;

determining a change in actual marker position for each marker on the face for each selected facial movement; and

graphing the change in actual marker position on the face for each selected facial movement.

27. A method for measuring and quantitating movements of a face, the method comprising the steps of:

centering pupils of the face at repose;

applying a marker to a plurality of facial zones including radix, philtrum, mentum, left supraorbital, right supraorbital, left infraorbital, right infraorbital, left oral commissure and right oral commissure zones;

positioning a calibration ruler proximate to a tip of a nose of the face by positioning means, said calibration ruler including at least one calibration unit;

applying a reference marker to the nose behind and partially above the calibration ruler so that the reference marker partially appears above the calibration ruler when the face is viewed head-on;

recording an image of the marked face at repose focused on the calibration ruler using a compact disc camera capable of capturing facial images on digitally based media;

using said compact disc camera and focusing on said calibration ruler, recording an image of the marked face at a plurality of selected facial movements including maximal brow lift, maximal tight eye closure, maximal smile, maximal frown and maximal mouth pucker;

viewing each facial image by using a computer having a computer screen;

juxtapositioning a grid to each facial image displayed on the computer screen so that a horizontal axis of the grid passes through lower limbs of a medial canthi of the facial image and a vertical axis of the grid passes through a perceived center of the bony nasal dorsum of the facial image, said grid including at least one grid unit;

recording location coordinates for each marker of each facial image based on the grid orientation;

calibrating each facial image by placing the grid proximate to the image of the calibration ruler of each facial image so as to determine the number of grid units per calibration unit for that facial image;

converting the location coordinates of each marker to actual positions on the face relative to a trans-canthal/trans-nasal origin formed by the medial canthi and the bony nasal dorsum of the facial image;

determining a change in actual marker position for each marker on the face for each selected facial movement; and

graphing the change in actual marker position on the face for each selected facial movement.

28. A method for measuring and quantitating movements of a face, the method comprising the steps of:

centering pupils of the face at repose;

applying a marker to a plurality of facial zones including radix, philtrum, mentum, left supraorbital, right supraorbital, left infraorbital, right infraorbital, left oral commissure and right oral commissure zones;

positioning a calibration ruler proximate to a tip of a nose of the face by positioning means;

applying a reference marker to the nose behind and partially above the calibration ruler so that the reference marker partially appears above the calibration ruler when the face is viewed head-on;

recording an image of the marked face at repose focused on the calibration ruler using a camera capable of capturing facial images on film for developing slides;

using said camera and focusing on said calibration ruler, recording an image of the marked face at a plurality of selected facial movements including maximal brow lift, maximal tight eye closure, maximal smile, maximal frown and maximal mouth pucker;

developing said facial images into slides;

projecting each facial image slide onto a digitizer board so that the calibration ruler is projected to its original size;

calibrating the digitizer board;

selectively aligning a data input device with each marker of each projected facial image, the data input device communicating with a digitizer computer program and the digitizer board to assign actual marker positions to each marker of each facial image;

using an adapted computer spreadsheet to determine a change in actual marker position for each marker on the face for each selected facial movement; and

graphing the change in actual marker position on the face for each selected facial movement.

29. A method for measuring and quantitating movements of a face, the method comprising the steps of:

centering pupils of the face at repose;

applying a marker to a plurality of facial zones including radix, philtrum, mentum, left supraorbital, right supraorbital, left infraorbital, right infraorbital, left oral commissure and right oral commissure zones;

positioning a calibration ruler proximate to a tip of a nose of the face by positioning means;

applying a reference marker to the nose behind and partially above the calibration ruler so that the reference marker partially appears above the calibration ruler when the face is viewed head-on;

recording an image of the marked face at repose focused on the calibration ruler using a camera capable of capturing facial images on digitally based media;

using said camera and focusing on said calibration ruler, recording an image of the marked face at a plurality of selected facial movements including maximal brow lift, maximal tight eye closure, maximal smile, maximal frown and maximal mouth pucker;

projecting each facial image onto a computer screen so that the calibration ruler is projected to its original size;

calibrating the computer screen;

selectively aligning a data input device with each marker of each projected facial image, the data input device communicating with a digitizer computer program and the computer screen to assign actual marker positions to each marker of each facial image;

using an adapted computer spreadsheet to determine a change in actual marker position for each marker on the face for each selected facial movement; and

graphing the change in actual marker position on the face for each selected facial movement.