Modeling shape, motion, and flexion of non-rigid 3D objects in a sequence of images

• US 20030072482A1
• Filed: 02/22/2001
• Published: 04/17/2003
• Est. Priority Date: 02/22/2001
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Information Technology Center America, Inc. (Mitsubishi Electric Corporation)
• Inventors: Brand, Matthew
• Granted Patent: US 7,006,683 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/579 from motion

Abstract

A method models a non-rigid three-dimensional object directly from a sequence of images. A shape of the object is represented as a matrix of 3D points, and a basis of possible deformations of the object is represented as a matrix of displacements of the 3D points. The matrices of 3D points and displacements forming a model of the object. Evidence for an optical flow is determined from image intensities in a local region near each 3D point. The evidence is factored into 3D rotation, translation, and deformation coefficients of the model to track the object in the video.

Claims

1. A method for modeling a non-rigid three-dimensional object directly from a sequence of images, comprising:

representing a shape of the object as a matrix of 3D points, and a basis of possible deformations of the object as a matrix of displacements of the 3D points, the matrices of 3D points and displacements forming a model of the object;

determining evidence for an optical flow from image intensities in a local region near each 3D point; and

factoring the evidence into 3D rotation, translation, and deformation coefficients of the model to track the object in the video.

2. The method of claim 1 wherein the evidence includes local spatial variation, temporal variation, and spatial gradients of image intensities in the local regions in each image of the sequence.

3. The method of claim 2 wherein the spatial variation at any point

4. The method of claim 1 wherein an orthographic projection of the model onto each image of the video is expressed by

P=R_d×

3(B<sub>3×

n</sub>+(C<sub>×

k</sub>{circle over (x)}I₃)D<sub>3k×

n</sub>)⊕

T_2×

,

where R is a rotation matrix, B is a shape matrix, I is an identity matrix, D is a deformation matrix, C is a flexion of all the deformations, and T a translation matrix.

5. The method of claim 4 Wherein the matrix of displacements includes a matrix of shape to handle scaling of the object in the video.

6. The method of claim 1 wherein the rotation, translation, and deformation are determined with a minimal number of inner product and least-squares operations to minimize information loss, and all least-squares calculations utilize elliptical error norms derived from the evidence.

7. The method of claim 1 wherein the optical flow determined from the evidence for local regions in the sequence of images is constrained to be globally consistent with the model.

8. The method of claim 1 wherein the model is derived directly from a generic model and residuals obtained by using the generic model to track the object in video using the evidence, rotation, translation, and deformation.