Method for detecting human body motion in frames of a video sequence

• US 5,930,379 A
• Filed: 06/16/1997
• Issued: 07/27/1999
• Est. Priority Date: 06/16/1997
• Status: Expired due to Term

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Digital Equipment Corporation (HP Inc.)
• Inventors: Rowley, Henry A., Rehg, James M.
• Primary Examiner(s): Couso, Jose L.
• Assistant Examiner(s): MEHTA, BHAVESH M
• Application Number:
  US08/876,603
• Time in Patent Office: 771 Days
• Field of Search: 382/103, 382/107, 382/155-159, 382/236, 382/224-228, 382/181, 382/190, 382/276, 382/286, 348/153-155, 345/419-420, 345/427-432, 345/328, 345/424, 345/473-474
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

In a computerized method, a moving object is detected in a sequence of frames of a video of a scene. Each of the frames includes a plurality of pixels representing measured light intensity values at specific locations in the scene. The pixels are organized in a regularized pattern in a memory. The object is modeled as a branched kinematic chain composed of links connected at joints. The frames are iteratively segmented by assigning groups of pixels having like pixel motion to individual links, while estimating motion parameters for the groups of pixels assigned to the individual links until the segmented pixels and their motion parameters converge and can be identified with the moving object as modeled by the kinematic chain.

Claims

1. A computerized method for detecting a moving object in a sequence of frames of a video of a scene, each frame including a plurality of pixels representing light intensity values at specific locations in the scene, comprising the steps of:

representing the object as a branched kinematic chain composed of links connected at joints;

iteratively segmenting frames by assigning groups of pixels to individual links, while estimating motion parameters for the groups of pixels assigned to the individual links until the segmented groups of pixels and the estimated motion parameters converge; and

identifying the groups of pixels as representing the moving object modeled by the branched kinematic chain.

2. The method of claim 1 wherein the moving object is a person, and the plurality of links model the limbs and torso of the person.

3. The method of claim 1 wherein each of the joints connecting the links has a predetermined degrees of freedom about which the links can be displaced.

4. The method of claim 1 wherein the links have predetermined relative lengths.

5. The method of claim 1 wherein the motion parameters include the relative angular displacement of groups of pixels assigned to links connected by a common joint.

6. The method of claim 1 wherein a first link is designated as the base of the kinematic chain.

7. The method of claim 6 further comprising:

describing a position of a second link as a coordinate transformation relative to the base.

8. The method of claim 7 further comprising:

describing a position of the base as a rotation and translation with respect to the scene.

9. The method of claim 1 further comprising:

modeling pixel motion between two temporally adjacent frames due to a change of a joint angle by a kinematic deformation function.

10. The method of claim 1 wherein the video is partitioned into a plurality of volumes, each volume includes a predetermined number of frames.

11. The method of claim 1 wherein the representing, segmenting, estimating, and identifying steps are performed separately on each of the plurality of volumes.

12. The method of claim 10 wherein a space-time pyramid is extracted from each volume.

13. The method of claim 12 wherein the space-time pyramid is extracted by downsampling the frames of each volume over time and space.

14. The method of claim 1 wherein the groups of pixels are classified for identification by a feed-forward neural network.

15. The method of claim 1 wherein the branched kinematic chain constrain how the links move relative to each other.

16. The method of claim 3 wherein the links move rigidly depending on the degrees of freedom at the joints.

17. The method of claim 1 further comprising:

associating a segmentation map and kinematic parameters which each link, the segmentation map identifying a particular group of pixels associated with the link, and the kinematic map describing the motion of the particular group of pixels.

18. The method of claim 17 wherein the segmentation map and the kinematic parameters are represented as a feature vector of the associated link.

19. The method of claim 17 wherein the segmentation map includes an array of likelihoods that the group of pixels is associated with the link.

20. The method of claim 17 wherein the segmentation map and the kinematic parameters of a particular link are iteratively estimated in alternating steps until the segmentation map and the kinematic parameters converge.

21. The method of claim 17 further comprising:

estimating the motion parameters using a deformation function F(P, x, ), where P are the kinematic parameters of a particular pixel at a location x and y.

22. The method of claim 1 wherein a shape of a particular link is determined from the motion of the associated group of pixels.

23. A computerized method for detecting a moving object in a sequence of frames of a video of a scene, each frame including a plurality of pixels representing light intensity values at specific locations in the scene, comprising the steps of:

modeling the object by a set of links and a set of dependent differential vector functions which constraint the motion of the links;

iteratively segmenting frames by assigning groups of pixels to individual links, while estimating motion parameters for the groups of pixels assigned to the individual links until the segmented groups of pixels and the estimated motion parameters converge; and

identifying the groups of pixels as representing the moving object as modeled by the set of links and vector functions.

24. The method of claim 23 wherein the set of links and the dependent differential vector functions are organized as a branched kinematic chain.