Motion capture from body mounted cameras

• US 8,786,680 B2
• Filed: 06/21/2011
• Issued: 07/22/2014
• Est. Priority Date: 06/21/2011
• Status: Active Grant

• Current Assignee: Carnegie Mellon University, Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Carnegie Mellon University, Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Shiratori, Takaaki, Park, Hyun Soo, Sigal, Leonid, Sheikh, Yaser, Hodgins, Jessica K.
• Primary Examiner(s): Anyikire, Chikaodili E
• Assistant Examiner(s): NASRI, MARYAM A
• Application Number:
  US13/165,619
• Publication Number:
  US 20120327194A1
• Time in Patent Office: 1,127 Days
• US Class Current: 348/47
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Body-mounted cameras are used to accurately reconstruct the motion of a subject. Outward-looking cameras are attached to the limbs of the subject, and the joint angles and root pose that define the subject'"'"'s configuration are estimated through a non-linear optimization, which can incorporate image matching error and temporal continuity of motion. Instrumentation of the environment is not required, allowing for motion capture over extended areas and in outdoor settings.

Claims

1. A method for capturing motion of a subject having a body with a plurality of articulating members movable about a plurality of joints, the method comprising:

attaching a plurality of cameras to the body of the subject, wherein at least some of the cameras are attached to the articulating members and oriented toward an external environment surrounding the subject;

recording an image sequence from each of the cameras while the subject performs an action;

analyzing, using a computer system, the image sequences recorded by each of the cameras, including images of the external environment from the cameras attached to the articulating members, to determine correlations between the images; and

based at least in part on the analysis of the image sequences, including the images of the external environment from the cameras attached to the articulating members, computing, using the computer system, a sequence of articulated poses of the body of the subject, the sequence of articulated poses being representative of the action.

2. The method of claim 1 further comprising:

rendering a computer-generated animation of an actor performing the action based at least in part on the computed sequence of articulated poses of the body of the subject.

3. The method of claim 1 wherein the external environment includes a plurality of objects that remain stationary while the subject performs the action.

4. The method of claim 3 wherein analyzing the image sequences includes generating a representation of three-dimensional scene geometry based on correlating visual features across multiple ones of the images in the recorded image sequences.

5. The method of claim 1 wherein analyzing the image sequences includes:

obtaining a model of three-dimensional scene geometry for the external environment from a source other than the recorded image sequences; and

using the model of the three-dimensional scene geometry to facilitate determination of correlations between the images.

6. The method of claim 1 wherein at least one of the cameras is a visible-light camera with a field of view subtending an angle of at least 150 degrees.

7. The method of claim 1 wherein each of the cameras has a field of view such that the fields of view of at least some of the cameras overlap when the subject'"'"'s body is in a resting position.

8. The method of claim 1 wherein the action performed by the subject includes performing a series of training motions during which different joints of the body of the subject are moved through a full range of motion.

9. The method of claim 1 further comprising:

recording a plurality of reference images of the external environment,

wherein analyzing the image sequences recorded by each of the cameras includes analyzing the reference images.

10. The method of claim 9 wherein at least one of the plurality of reference images is recorded using a camera that is not attached to the body of the subject.

11. The method of claim 1 wherein attaching the plurality of cameras includes attaching at least one camera to an article of clothing to be worn by the subject.

12. The method of claim 1 wherein attaching the plurality of cameras includes attaching at least one camera to the subject using a strap.

13. The method of claim 1 wherein computing the sequence of articulated poses includes defining a root for the body and determining a position of at least one of the plurality of joints relative to the root.

14. The method of claim 13 wherein computing the sequence of articulated poses further includes determining an absolute position and orientation for the root within the external environment.

15. A system for capturing motion of a subject having a body with a plurality of articulating members, the system comprising:

a data input port configured to receive a plurality of image sequences captured by a plurality of cameras worn by a subject while performing an action, wherein at least some of the cameras are attached to the articulating members of the body of the subject and oriented toward an external environment surrounding the subject; and

a processor configured to;

analyze the image sequences received from each of the plurality of cameras, including images of the external environment received from the cameras attached to the articulating members of the body, to determine correlations between the images; and

compute, based at least in part on the analysis of the image sequences, including the images of the external environment, a sequence of articulated poses of the body of the subject, the sequence of articulated poses being representative of the action.

16. The system of claim 15 wherein the processor is further configured such that computing the sequence of articulated poses includes:

applying a structure-from-motion algorithm to the image sequences to determine a geometric representation of a three-dimensional scene and an initial estimate of camera poses; and

applying a reprojection optimization to refine the initial estimate of the camera poses.

17. The system of claim 16 wherein the processor is further configured such that computing the sequence of articulated poses further includes:

computing a skeleton based on images from the image sequence associated with a range-of-motion exercise performed by the subject.

18. A system for capturing motion of a subject having a body with a plurality of articulating members, the system comprising:

a plurality of cameras adapted to be wearable by a subject while performing an activity, the plurality of cameras including at least two cameras wearable on the articulating members of the body of the subject and orientable toward an external environment surrounding the subject;

a storage medium configured to receive and store a plurality of image sequences captured by the plurality of cameras, the image sequences including images of the external environment captured while the subject performed the activity; and

a processor coupled to the storage medium and configured to;

analyze the image sequences including the images of the external environment to determine correlations between the images; and

compute, based at least in part on the analysis of the image sequences, including the images of the external environment, a sequence of articulated poses of the body of the subject, the sequence of articulated poses being representative of the action.

19. The system of claim 18 wherein the cameras are operable to capture images at a rate of at least 60 frames per second.

20. The system of claim 18 wherein at least one of the cameras is a visible-light camera with a field of view subtending an angle of at least 150 degrees.