Method and apparatus for tracking moving objects in a sequence of two-dimensional images using a dynamic layered representation

• US 6,826,292 B1
• Filed: 06/23/2000
• Issued: 11/30/2004
• Est. Priority Date: 06/23/2000
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Kumar, Rakesh, Tao, Hai, Sawhney, Harpreet Singh
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/603,666
• Time in Patent Office: 1,621 Days
• Field of Search: 382/103, 382/107, 382/108, 382/276, 382/293, 382/305, 382/236, 382/173, 382/206, 382/232, 345/16, 345/3.2, 345/721, 345/719, 345/328, 345/432, 345/427, 345/430, 348/155, 348/335, 348/584, 348/700, 375/240.16
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20116 Active contour; Active surface; Snakes
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

A system that tracks one or more moving objects in a sequence of video images employs a dynamic layer representation to represent the objects that are being tracked. The system concurrently estimates three components of the dynamic layer representation—layer segmentation, motion, and appearance—over time in a maximum a posteriori (MAP) framework. In order to enforce a global shape constraint and to maintain the layer segmentation over time, the subject invention imposes a prior constraint on parametric segmentation. In addition, the system uses a generalized Expectation-Maximization (EM) algorithm to compute an optimal solution. The system uses an object state that consists of representations of motion, appearance and ownership masks. The system applies a constant appearance model across multiple images in the video stream.

Claims

1. Apparatus for tracking one or more moving objects in a sequence of video images comprising:

a moving object detection processor that identifies at least one area in a plurality of images selected from the sequence of video images, the at least one identified area representing moving objects; and

a tracking system comprising;

means forsegmenting the at least one identified area in the images;

means for determining at least one respective motion parameter for the at least one identified area in the images; and

means for determining a respective appearance for the at least one identified area in the images by image wariping of pixel in teh at least one identified area based on the at least one determined respective motion parameter.

2. Apparatus according to claim 1, wherein the tracking system further comprises means for calculating an ownership parameter for the at least one identified area in the images, the ownership parameter being a funtion of the segmentation, the motion parameter and the appearance.

5. Apparatus according to claim 1, wherein the means for determining the at least one respective motion parameter Includes means for determining an estimated translation velocity and an estimated rotation velocity for the at least one Identified area.

6. Apparatus according to claim 1, wherein the means for determining the at least one respective appearance parameter determines a plurality of intensity measurements in the identified area.

7. Apparatus according to claim 1, wherein the tracking system includes means for maintaining a description of each moving object in the sequence of video images as a respective layer in a dynamic layer representation, each layer having a segmentation constraint, a motion constraint and an appearance constraint and the tracking system further includes means for associating the at least one identified area with at least-one respective layer by concurrently matching the segmentation of the identified area to the segmentation constraint, the at least one velocity parameter of the identified area to the velocity constraint and the at least one appearance parameter of the area to the appearance constraint.

8. Apparatus according to claim 7, wherein the means for maintaining the description of each moving object Includes means for concurrently estimating layer motion, segmentation and appearance over time using a maximum a posteriori (MAP) framework.

9. Apparatus according to claim 8, wherein the means for maintaining the description of each moving object further includes means for applying a generalized Expectation-Maximization (EM) algorithm to compute an optimal solution.

10. Apparatus according to claim 9, wherein the means for maintaining the description of each moving object further includes means for applying a constant appearance model across multiple images in the sequence of video images.

3. Apparatus for tracking one or more moving objects in a sequence of video images comprising:

a moving object detection processor that identifies at least one area in a plurality of images selected from the sequence of video images, the at least one identified area representing moving objects; and

a tracking system compirising;

means for segmenting the at least one identified area in the images, including means for fitting an elliptical shape model to the identified area;

means for determining at least one respective motion parameter for the at least one identified area in the images; and

means for determining a respective appearance for the at least one identified area in the images.

4. Apparatus according to claim 3, wherein the elliptical shape model corresponds to the distribution:

11. A method for tracking one or more moving objects in a sequence of video images comprising:

the step of identifying at least one area in a plurality of images selected from the sequence of video images, the at least one identified area representing moving objects; and

the step of tracking the at least one identified area comprising the steps of;

segmenting the at least one identified area In the images;

determining at least one respective motion parameter for the at least one identified area in the imnages; and

determining a respective appearance for the at least one identified area in the images by image warping of pixels in the at least one identifled area based on the at least one determined resoective motion parameter.

12. A method according to claim 11, wherein the tracking step further comprises the act of calculating an ownership parameter for the at least one identified area in the images, the ownership parameter being a function of the segmentation, the at least one motion parameter and the appearance.

15. A method according to claim 11, wherein the step of determining the at least one respective motion parameter includes the acts of determining an estimated translation velocity and an estimated rotation velocity for the at least one identified area.

16. A method according to claim 11, wherein the step of determining the at least one respective appearance parameter includes the act of determining a plurality of intensity measurements in the identified area.

17. A method according to claim 11, further including the step of maintaining a description of each moving object in the sequence of video images as a respective layer in a dynamic layer representation, each layer having a segmentation constraint, a motion constraint and an appearance constraint, and the method further includes the step of associating the at least one identified area with at least one respective layer by concurrently matching the segmentation of the identified area to the segmentation constraint, the at least one velocity parameter of the identified area to the velocity constraint and the at least one appearance parameter of the area to the appearance constraint.

18. A method according to claim 17, wherein the step of maintaining the description of each moving object includes the act of concurrently estimating layer motion, segmentation and appearance over time using a maximum a posteriori (MAP) framework.

19. A method according to claim 18, wherein the step of maintaining the description of each moving object further Includes the act of applying a generalized Expectation-Maximization (EM) algorithm to compute an optimal solution.

20. A method according to claim 19, wherein the step of maintaining the description of each moving object further includes the act of applying a constant appearance model across multiple Images in the sequence of video images.

13. A method for tracking one or more moving objects in a sequence of video images comprising:

the step of identlfying at least one area in a plurality Of r mages selected from the sequence of video images, the at least one identified area representina moving objects; and

the step of tracking the at least one identified area comprising the steps of;

segmenting the at least one identified area in the images, ineludes includino the act of fitting an elliptical shape model to the identified area;

determining at least one respective motion parameter for the at least one identified area in the images; and

determining a respective appearance for the at least one identified area in the images.

14. A method according to claim 13, wherein the elliptical shape model corresponds to the distribution;

21. An article of manufacture comprising a carrier including computer software that causes a computer to perform a method for tracking one or more moving objects in a sequence of video images, the method comprising:

the step of identifying at least one area in a plurality of images selected from the sequence of video images, the at least one identified area representing moving objects; and

the step of tracking the at least one identified area comprising the steps of;

segmenting the at least one identified area in the Images;

determining at least one respective motion parameter for the at least one identified area in the Images; and

determining a respective appearance for the at least one Identified area in the images by imaae warping of pixels in the at least one identified area based on the at least one determined respective motion parameter.

22. An article of manufacture according to claim 21, wherein, as a part of the tracking step, the computer program causes the computer to perform the act of calculating an ownership parameter for the at least one identified area in the Images;

the ownership parameter being a function of the segmentation, the at least one motion parameter and the appearance.

24. An article of manufacture according to claim 21, wherein the computer program further causes the computer to maintain a description of each moving object in the sequence of video images as a respective layer in a dynamic layer representation, each layer having a segmentation constraint, a motion constraint and an appearance constraint, and further causes the computer to associate the at least one identified area with at least one respective layer by concurrently matching the segmentation of the identified area to the segmentation constraint, the at least one velocity parameter of the identified area to the velocity constraint and the at least one appearance parameter of the area to the appearance constraint.

23. An article of manufacture Icomprising a carrier including comouter software that causes a computer to perform a method for tracking one or more moving objects in a sequence of video images, the method comprising;

the step of identifying at least one area in a pluralty of images select from the sequence of video images, the at least one identified area representing movina objects; and

the step of tracking the at least one identified area cotnorisino the steps of;

segmenting the at least one identified area in the images, including the act of fitting an elliptical shape model to the identified areas determining at least one respective motion parameter for the at least one identified area in the images; and

determnining a respective appearance for the atleast one identifled area in the images.