Apparatus and method for tracking facial motion through a sequence of images
First Claim
1. A method for estimating movements of a face and a facial feature of a human appearing in a sequence of images recorded over time, comprising the steps of:
- (a) acquiring with a processor a first image and a second image from the sequence of images, the first image being recorded at a first instant in time and the second image being recorded at a second instant in time;
(b) locating with the processor a first region of the first image in which the face is located and a second region of the first image in which the facial feature is located;
(c) computing with the processor a first transformation to estimate a movement of the face between the first region in the first image and a first region in the second image;
(d) forming a third image using the processor by warping the second image with the first transformation, said forming step eliminating, in the third image, the movement of the face between the first region in the first image and the first region in the second image so as to isolate, in the third image, a movement of the facial feature between the second region in the first image and a second region in the second image;
(e) computing with the processor a second transformation to estimate a movement of the facial feature between the second region in the first image and a region in the third image; and
(f) combining with the processor the first transformation and the second transformation to estimate the movement of the facial feature between the second region in the first image and the second region in the second image.
7 Assignments
0 Petitions
Accused Products
Abstract
A system tracks human head and facial features over time by analyzing a sequence of images. The system provides descriptions of motion of both head and facial features between two image frames. These descriptions of motion are further analyzed by the system to recognize facial movement and expression. The system analyzes motion between two images using parameterized models of image motion. Initially, a first image in a sequence of images is segmented into a face region and a plurality of facial feature regions. A planar model is used to recover motion parameters that estimate motion between the segmented face region in the first image and a second image in the sequence of images. The second image is warped or shifted back towards the first image using the estimated motion parameters of the planar model, in order to model the facial features relative to the first image. An affine model and an affine model with curvature are used to recover motion parameters that estimate the image motion between the segmented facial feature regions and the warped second image. The recovered motion parameters of the facial feature regions represent the relative motions of the facial features between the first image and the warped image. The face region in the second image is tracked using the recovered motion parameters of the face region. The facial feature regions in the second image are tracked using both the recovered motion parameters for the face region and the motion parameters for the facial feature regions. The parameters describing the motion of the face and facial features are filtered to derive mid-level predicates that define facial gestures occurring between the two images. These mid-level predicates are evaluated over time to determine facial expression and gestures occurring in the image sequence.
543 Citations
26 Claims
-
1. A method for estimating movements of a face and a facial feature of a human appearing in a sequence of images recorded over time, comprising the steps of:
-
(a) acquiring with a processor a first image and a second image from the sequence of images, the first image being recorded at a first instant in time and the second image being recorded at a second instant in time; (b) locating with the processor a first region of the first image in which the face is located and a second region of the first image in which the facial feature is located; (c) computing with the processor a first transformation to estimate a movement of the face between the first region in the first image and a first region in the second image; (d) forming a third image using the processor by warping the second image with the first transformation, said forming step eliminating, in the third image, the movement of the face between the first region in the first image and the first region in the second image so as to isolate, in the third image, a movement of the facial feature between the second region in the first image and a second region in the second image; (e) computing with the processor a second transformation to estimate a movement of the facial feature between the second region in the first image and a region in the third image; and (f) combining with the processor the first transformation and the second transformation to estimate the movement of the facial feature between the second region in the first image and the second region in the second image.
-
-
2. A system for estimating movements of a first object and a second object appearing in a sequence of images recorded over time, comprising:
-
a sequence manager for acquiring with a processor a first image and a second image from the sequence of images, the first image being recorded at a first instant in time and the second image being recorded at a second instant in time; an image segmentation system for locating with the processor a first region of the first image in which the first object is located and a second region of the first image in which the second object is located; and a motion estimation system including; first means for computing with the processor a first transformation that estimates a movement of the first object between the first region in the first image and a first region in the second image; and second means for computing with the processor a second transformation that estimates a movement of the second object relative to the movement of the first object between the first image and the second image with the first transformation, wherein said second means for computing comprises; means for forming a third image using the processor by warping the second image with the first transformation, said forming means eliminating, in the third image, the movement of the first object between the first region in the first image and the first region in the second image so as to isolate, in the third image, a movement of the second object between the second region in the first image and a second region in the second image; and means for combining with the processor the first transformation and the second transformation to estimate the movement of the second object between the second region in the first image and the second region in the second image. - View Dependent Claims (23, 24, 25, 26)
-
-
3. A method for estimating movements of a first object and a second object appearing in a sequence of images recorded over time, comprising the steps of:
-
(a) acquiring with a processor a first image and a second image from the sequence of images, the first image being recorded at a first instant in time and the second image being recorded at a second instant in time; (b) locating with the processor a first region of the first image in which the first object is located and a second region of the first image in which the second object is located; (c) computing with the processor a first transformation that estimates a movement of the first object between the first region in the first image and a first region in the second image; (d) computing with the processor a second transformation that estimates a movement of the second object relative to the movement of the first object between the first image and the second image with the first transformation, wherein step (d) comprises the steps of; forming a third image using the processor by warping the second image with the first transformation, said forming step eliminating, in the third image, the movement of the first object between the first region in the first image and the first region in the second image so as to isolate, in the third image, a movement of the second object between the second region in the first image and a second region in the second image; computing with the processor a second transformation to estimate a movement of the second object between the second region in the first image and a region in the third image; and combining with the processor the first transformation and the second transformation to estimate the movement of the second object between the second region in the first image and the second region in the second image. - View Dependent Claims (4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22)
-
Specification