Real time markerless motion tracking using linked kinematic chains

• US 7,257,237 B1
• Filed: 03/04/2004
• Issued: 08/14/2007
• Est. Priority Date: 03/07/2003
• Status: Active Grant

• Current Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Original Assignee: Sandia Corporation (Martin Marietta Corporation)
• Inventors: Luck, Jason P., Small, Daniel E.
• Primary Examiner(s): Werner; Brian P.
• Assistant Examiner(s): RADKIEWICZ, JARED
• Application Number:
  US10/794,331
• Time in Patent Office: 1,258 Days
• Field of Search: 348/169, 382/103, 382/154
• US Class Current: 382/103
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/4528 Joints A61B5/4533, A61B5/4538 take precedence
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A markerless method is described for tracking the motion of subjects in a three dimensional environment using a model based on linked kinematic chains. The invention is suitable for tracking robotic, animal or human subjects in real-time using a single computer with inexpensive video equipment, and does not require the use of markers or specialized clothing. A simple model of rigid linked segments is constructed of the subject and tracked using three dimensional volumetric data collected by a multiple camera video imaging system. A physics based method is then used to compute forces to align the model with subsequent volumetric data sets in real-time. The method is able to handle occlusion of segments and accommodates joint limits, velocity constraints, and collision constraints and provides for error recovery. The method further provides for elimination of singularities in Jacobian based calculations, which has been problematic in alternative methods.

Claims

1. A Markerless method for tracking a subject in a workspace comprising:

providing a video imaging system comprising;

a plurality of video cameras simultaneously viewing the subject in said workspace;

means for digitally acquiring images output from said video cameras; and

,

means for digitally processing said images;

calibrating said video cameras and said workspace, said workspace further comprising a three dimensional array of voxels;

providing a model template for said subject comprising a plurality of rigid segments linked by joints;

initializing a model of said subject comprising;

acquiring a first volumetric data set from said video imaging system;

said first volumetric data set comprising a listing of said voxels occupied by said subject;

dimensioning the rigid segments of the model template by fitting said rigid segments to said first volumetric data set;

acquiring a second volumetric data set from said video imaging system, said second volumetric data set further comprising listing of said voxels occupied by said subject;

aligning the model to said second volumetric data set, said aligning further comprising calculating in three dimensions a plurality of forces acting on said model, applying said plurality of forces to said model, said plurality of forces causing a displacement of the model;

wherein aligning said model to said second volumetric data set further comprises;

(a) projecting said occupied voxels from said second data set onto said rigid segments of the model, wherein each of said occupied voxels is projected onto each of the rigid segments nearest each of said occupied voxels;

(b) calculating a force exerted on each of said rigid segments by each of said occupied voxels projected onto each of said rigid segments;

(c) combining the force exerted on each of said rigid segments into an equivalent force exerted on each of said rigid segments;

(d) calculating a torque exerted on each of said rigid segments, about each of said joints;

(e) calculating a translational acceleration resultant from said equivalent force, acting on each of said rigid segments;

(f) calculating a rotational acceleration resultant from said torque, exerted on each of said rigid segments, about each of said joints;

(g) determining an incremental displacement of each of said rigid segments resulting from the calculated accelerations;

(h) displacing each rigid segment by the calculated incremental displacement;

(i) computing a distance between each of said occupied voxels and each of said rigid segments each of said occupied voxels is projected onto;

(j) comparing the computed distance to a threshold value; and

(k) repeating steps (b) through (j) until said computed distance is less than the threshold value;

whereby said model segments are iteratively aligned to the second data set; and

,

repeating the steps of acquiring a second volumetric data set and aligning the model to said second volumetric data set ad infinitum.

2. The method of claim 1 wherein calculating a force on each of said rigid segments further comprises:

providing said model of said subject further comprising;

a first segment;

a second segment having a first end and an opposed second end, said first end of said second segment joined to said first segment by a first rotatable joint;

a third segment having a third end and an opposed fourth end, said third end of said third segment joined to said second end of said second segment by a second rotatable joint, said first, second and third segments being rigid segments;

a first joint model of said first rotatable joint, said first joint model further comprising two joint elements, each of said two joint elements having one degree of freedom and an included angle;

a second joint model of said second rotatable joint said second joint model further comprising four joint elements, each of said four joint elements having one degree of freedom and an included angle;

performing a coordinate transformation of said second segment, wherein said included angle of said two joint elements of said first joint model is equal to zero;

applying a Jacobian to calculate said force exerted on said second segment by each of said occupied voxels projected onto said second segment;

selecting two of said four joint elements comprising said second joint model, setting said included angle of the two selected joint elements in said second joint model to zero;

applying a Jacobian to calculate said force exerted on said third segment by each of said occupied voxels projected onto said third segment;

whereby singularities are avoided in the Jacobian calculations.