Moving Target Detection in the Presence of Parallax

• US 20080089556A1
• Filed: 06/15/2007
• Published: 04/17/2008
• Est. Priority Date: 06/20/2006
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Salgian, Garbis, Samarasekera, Supun, Han, Feng, Kumar, Rakesh, Xiao, Jiangjian, Bergen, James
• Granted Patent: US 8,340,349 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30212 Military
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes

Abstract

A method for detecting a moving target is disclosed that receives a plurality of images from at least one camera; receives a measurement of scale from one of a measurement device and a second camera; calculates the pose of the at least one camera over time based on the plurality of images and the measurement of scale; selects a reference image and an inspection image from the plurality of images of the at least one camera; and detects a moving target from the reference image and the inspection image based on the orientation of corresponding portions in the reference image and the inspection image relative to a location of an epipolar direction common to the reference image and the inspection image; and displays any detected moving target on a display. The measurement of scale can derived from a second camera or, for example, a wheel odometer. The method can also detect moving targets by combining the above epipolar method with a method based on changes in depth between the inspection image and the reference image and based on changes in flow between the inspection image and the reference image.

Claims

1. A method for detecting a moving target, comprising the steps of:

receiving a plurality of images from at least one camera;

receiving a measurement of scale from one of a measurement device and a second camera;

calculating the pose of the at least one camera over time based on the plurality of images and the measurement of scale;

selecting a reference image and an inspection image from the plurality of images of the at least one camera;

detecting a moving target from the reference image and the inspection image based on the orientation of corresponding portions in the reference image and the inspection image relative to a location of an epipolar direction common to the reference image and the inspection image; and

displaying any detected moving target on a display.

2. The method of claim 1, wherein the step of detecting a moving target further includes the step of producing an image based on changes in depth between the inspection image and the reference image and based on changes in flow between the inspection image and the reference image.

3. The method of claim 1, wherein the measurement of scale is derived from the second camera.

4. The method of claim 1, wherein the measurement of scale is derived from a wheel odometer.

5. The method of claim 2, further comprising the step of confirming the presence of the moving target by applying an appearance classifier to a portion of the reference image containing a candidate moving target.

6. The method of claim 5, further comprising the step of applying a pose filter to a pose determined by said step of calculating the pose.

7. The method of claim 6, wherein the pose is determined by visual odometry.

8. The method of claim 7, further comprising the step of receiving a data stream from an IMU.

9. The method of claim 7, further comprising the step of receiving a data stream from a GPS receiver.

10. The method of claim 8, further comprising the step synchronizing the data stream from the IMU with the plurality of images.

11. The method of claim 10, further comprising the step of superimposing a viewer on the reference image so as to highlight any detected moving targets.

12. The method of claim 1, wherein said step of detecting a candidate moving target based on the orientation of corresponding portions in the reference image and the inspection image relative to a location of an epipolar direction further comprises the steps of:

eliminating an image motion component of the inspection image due to camera rotation;

computing the optical flow between the reference image and the inspection image that was compensated for camera rotation;

determining the epipole location and direction from the inspection image and the reference image;

comparing a flow vector of a pixel in the inspection image with the direction of the epipole; and

;

labeling the pixel as belonging to an independently moving object when the difference in direction between the flow vector and the direction of the epipole is above a predetermined threshold; and

labeling the pixel as belonging to fixed object when the difference in direction between the flow vector and the direction of the epipole is equal to or below the predetermined threshold.

13. The method of claim 2, wherein said step of producing an image based on changes in depth between the inspection image and the reference image and based on changes in flow between the inspection image and the reference image further comprises the steps of:

determining the depth of each pixel in the reference image and the inspection image;

warping the inspection image into the reference view of the reference image to produce a 3D warped image;

producing a 3D error image that is the difference between the 3D warped image and the reference image;

computing the optical flow between the reference image and the inspection image to produce a flow warping image;

producing a flow error image that is the difference between the flow warping image and the reference image; and

producing a second order residual difference image by subtracting the flow error image from the 3D error image; and

labeling any illuminated pixels in the second order residual difference image as belonging to an independently moving object.

14. An apparatus for detecting a moving target, comprising:

at least one camera for receiving a plurality of images;

means for receiving a measurement of scale from one of a measurement device and a second camera;

a processor communicatively connected to said at least one camera and said means for receiving a measurement of scale for calculating the pose of the at least one camera based on the plurality of images;

for selecting a reference image and an inspection image from the plurality of images of the at least one camera; and

for detecting a moving target from the reference image and the inspection image based on the orientation of corresponding portions in the reference image and the inspection image relative to a location of an epipolar direction common to the reference image and the inspection image; and

a viewer for displaying any detected moving target.

15. The apparatus of claim 14, wherein said processor further detect a moving target by producing an image based on changes in depth between the inspection image and the reference image and based on changes in flow between the inspection image and the reference image.

16. The apparatus of claim 14, wherein the measurement of scale is derived from the second camera.

17. The apparatus of claim 14, wherein the measurement of scale is derived from a wheel odometer.

18. The apparatus of claim of claim 15, wherein said processor confirms the presence of the moving target by applying an appearance classifier to a portion of the reference image containing a candidate moving target.

19. The apparatus of claim 18, wherein said processor further applies a pose filter to a pose.

20. The apparatus of claim 19, wherein the pose is determined by visual odometry.

21. The apparatus of claim 19, further comprising an IMU for sending measurements of orientation to said processor.

22. The apparatus of claim 20, further comprising a GPS receiver for sending a measurement of position to said processor.

23. A computer-readable medium carrying one or more sequences for detecting a moving target, wherein execution of the one of more sequences of instructions by one or more processors causes the one or more processors to perform the steps of:

receiving a plurality of images from at least one camera;

receiving a measurement of scale from one of a measuring device and a second camera;

calculating the pose of the at least one camera based on the plurality of images and the measurement of scale;

selecting a reference image and an inspection image from the plurality of images of the at least one camera;

detecting a moving target from the reference image and the inspection image based on the orientation of corresponding portions in the reference image and the inspection image relative to a location of an epipolar direction common to the reference image and the inspection image; and

displaying any detected moving target on a display.

24. The computer-readable medium of claim 23, wherein the step of detecting a moving target further includes the step of producing an image based on changes in depth between the inspection image and the reference image and based on changes in flow between the inspection image and the reference image.

25. The computer-readable medium of claim 24, wherein the measurement of scale is derived from the second camera.

26. The computer-readable medium of claim 24, wherein the measurement of scale is derived from a wheel odometer.

27. The computer-readable medium of claim 24, further comprising the step of confirming the presence of the moving target by applying an appearance classifier to a portion of the reference image containing a candidate moving target.

28. The computer-readable medium of claim 24, wherein said step of detecting a candidate moving target based on the orientation of corresponding portions in the reference image and the inspection image relative to a location of an epipolar direction further comprises the steps of:

eliminating an image motion component of the inspection image due to camera rotation;

computing the optical flow between the reference image and the inspection image that was compensated for camera rotation;

determining the epipole location and direction from the inspection image and the reference image;

comparing a flow vector of a pixel in the inspection image with the direction of the epipole; and

;

labeling the pixel as belonging to an independently moving object when the difference in direction between the flow vector and the direction of the epipole is above a predetermined threshold; and

labeling the pixel as belonging to fixed object when the difference in direction between the flow vector and the direction of the epipole is equal to or below the predetermined threshold.

29. The computer-readable medium of claim 24, wherein said step of producing an image based on changes in depth between the inspection image and the reference image and based on changes in flow between the inspection image and the reference image further comprises the steps of:

determining the depth of each pixel in the reference image and the inspection image;

warping the inspection image into the reference view of the reference image to produce a 3D warped image;

producing a 3D error image that is the difference between the 3D warped image and the reference image;

computing the optical flow between the reference image and the inspection image to produce a flow warping image;

producing a flow error image that is the difference between the flow warping image and the reference image; and

producing a second order residual difference image by subtracting the flow error image from the 3D error image; and

labeling any illuminated pixels in the second order residual difference image as belonging to an independently moving object.