Method and apparatus for detecting moving objects in two- and three-dimensional scenes

• US 6,049,619 A
• Filed: 02/10/1997
• Issued: 04/11/2000
• Est. Priority Date: 02/12/1996
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Anandan, Padmanabhan, Irani, Michal
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): Wu, Jingge
• Application Number:
  US08/797,911
• Time in Patent Office: 1,156 Days
• Field of Search: 382/103, 382/104, 382/106, 382/107, 382/294, 348/416
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

A method and apparatus for detecting moving objects in both two-dimensional and three-dimensional scenes. The method repetitively applies a two-dimensional transformation to a plurality of images representing a scene to identify misaligned regions within the images. Any residual motion represented by the misaligned regions that may be classified as a moving object within the scene is further processed by a three-dimensional technique that removes parallax motion from the residual motion. The result is motion contained in an epipolar flow field which is only due to a moving object within the scene.

Claims

1. A method for detecting object motion within a sequence of images representing a scene, comprising:

(a) selecting a plurality of images from said sequence of images;

(b) applying a two dimensional transformation to said plurality of images to align stationery regions of said plurality of images and identify a misaligned region of said plurality of images;

(c) computing a parallax rigidity constraint that identifies a parallax motion component of the misaligned region; and

(d) processing said parallax motion component to detect a moving object within the scene.

2. The method of claim 1 wherein step (b) further comprises the steps of:

(a'"'"') applying a two dimensional transformation to said plurality of images to align regions of said plurality of images and identify a plurality of misaligned regions of said plurality of images;

(b'"'"') segmenting said misaligned regions to produce segmented regions;

(c'"'"') iteratively aligning and segmenting the misaligned regions until a remaining misaligned region fulfills a criteria that identifies the remaining misaligned region as residual motion within said plurality of images.

3. The method of claim 2 wherein said segmenting step forms a layer containing said segmented regions.

4. The method of claim 3 wherein the segmenting step uses a segmentation mask to track the segmented regions over time.

5. The method of claim 2 wherein step (c'"'"') further comprises the step of:

determining if at least one of said plurality of misaligned regions is a moving object by determining if the misaligned region is consistent with the segmented regions.

6. The method of claim 5 wherein a misaligned region is deemed inconsistent with the segmented region if the misaligned region produces discontinuities in the two-dimensional motion everywhere along a boundary of the misaligned region.

7. The method of claim 5 further comprising the step of:

determining whether the misaligned region contains depth discontinuities that identify the misaligned region as a moving object.

8. The method of claim 1 wherein the processing step (d) further comprises the step of:

identifying a portion of the parallax motion component that does not conform to the parallax rigidity constraint as the moving object.

9. Apparatus for detecting object motion within a sequence of images representing a scene comprising:

means for selecting a plurality of images from said sequence of images;

means for applying a two-dimensional transformation to said plurality of images to align regions of said plurality of images and identify a misaligned region of said plurality of images;

means for computing a parallax rigidity constraint that identifies a parallax motion component of the misaligned region; and

means for processing said parallax motion component to detect a moving object within the scene.

10. The apparatus of claim 9 wherein said means for applying a two-dimensional transformation is iteratively applied to the plurality of images by segmenting the misaligned regions from the aligned regions to form a plurality of layers and aligning each layer with each other layer to identify a further misaligned regions and further layers until the layers cannot be aligned.

11. The apparatus of claim 10 wherein said means for processing said parallax motion component further comprises:

means for identifying a portion of the parallax motion component that violates the parallax rigidity constraint as the moving object within the scene.

12. A computer readable medium having stored thereon a plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the steps of:

(a) selecting a plurality of images from said sequence of images;

(b) applying a two dimensional transformation to said plurality of images to align stationery regions of said plurality of images and identify a misaligned region of said plurality of images;

(c) computing a parallax rigidity constraint that identifies a parallax motion component of the misaligned region; and

(d) processing said parallax motion component to detect a moving object within the scene.

13. The method of claim 12 wherein step (b) further comprises the steps of:

(a'"'"') applying a two dimensional transformation to said plurality of images to align regions of said plurality of images and identify a plurality of misaligned regions of said plurality of images;

(b'"'"') segmenting said misaligned regions to produce segmented regions;

(c'"'"') iteratively aligning and segmenting the misaligned regions until a remaining misaligned region fulfills a criteria that identifies the remaining misaligned region as motion within said plurality of images.

14. The method of claim 13 wherein said segmenting step forms a layer containing said segmented regions.

15. The method of claim 14 wherein the segmenting step uses a segmentation mask to track the segmented regions over time.

16. The method of claim 13 wherein step (c'"'"') further comprises the step of:

determining if at least one of said plurality of misaligned regions is a moving object by determining if the misaligned region is consistent with the segmented regions.

17. The method of claim 16 wherein a misaligned region is deemed inconsistent with the segmented region if the misaligned region produces discontinuities in the two-dimensional motion everywhere along a boundary of the misaligned region.

18. The method of claim 16 further comprising the step of:

determining whether the misaligned region contains depth discontinuities that identify the misaligned region as a moving object.

19. The method of claim 12 wherein the processing step (d) further comprises the step of:

identifying a portion of the parallax motion component that does not conform to the parallax rigidity constraint as the moving object.