Moving object detection using low illumination depth capable computer vision

• US 7,366,325 B2
• Filed: 10/12/2004
• Issued: 04/29/2008
• Est. Priority Date: 10/09/2003
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Liu, Xia, Fujimura, Kikuo
• Primary Examiner(s): Mehta; Bhavesh M.
• Assistant Examiner(s): Strege; John B
• Application Number:
  US10/964,299
• Publication Number:
  US 20050100192A1
• Time in Patent Office: 1,295 Days
• Field of Search: 382/104, 382/106, 382/154, 382/181, 382/225, 34042622- 33, 340435-437, 340/901, 340/907, 701301-302, 348/169
• US Class Current: 382/104
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Moving object detection is based on low illumination image data that includes distance or depth information. The vision system operates on a platform with a dominant translational motion and with a small amount of rotational motion. Detection of moving objects whose motions are not consistent with the movement of the background is complementary to shape-based approaches. For low illumination computer-based vision assistance a two-stage technique is used for simultaneous and subsequent frame blob correspondence. Using average scene disparity, motion is detected without explicit ego-motion calculation. These techniques make use of characteristics of infrared sensitive video data, in which heat emitting objects appear as hotspots.

Claims

1. A computer vision system for detecting moving objects from a moving platform, the system comprising:

an imaging subsystem for capturing scene image data from the moving platform at a plurality of times, the image data comprising information for deriving a depth measure for objects in the scene, the imaging subsystem for operation under low illumination conditions;

a computing subsystem communicatively coupled to the imaging subsystem for receiving the image data and for providing motion detection data indicating the detection of independently moving objects within the scene, the computing subsystem configurable to process the image data to determine three-dimensional coordinates for points corresponding to a plurality of objects in the scene and further comprising;

an average scene disparity module configurable to calculate an average scene disparity from changes in three-dimensional coordinates for a first subset of objects in the scene likely to be stationary;

an estimated position shift module configurable to calculate an estimated position shift for each object in a second subset of objects in the scene based at least in part on the average scene disparity;

an actual position shift module configurable to calculate an actual position shift for each object in the second subset of objects in the scene based in part on the three-dimensional coordinates for said second subset of objects;

a position error module configurable to calculate a position error for each object in the second subset of objects in the scene based, at least in part, on the difference between the estimated position shift and the actual position shift for each object; and

an object detection module configurable to detect at least one moving object from the objects in the second subset of objects based, at least in part, on a comparison between the position error for said at least one object and an empirically determined maximum threshold value; and

an output subsystem coupled to the computing subsystem for receiving the motion detection data and for providing a motion detection signal in response to the detection of one or more independently moving objects.

2. The computer vision system of claim 1, wherein the imaging subsystem further comprises a plurality of infrared-sensitive cameras.

3. The computer vision system of claim 1, wherein the moving platform is a vehicle comprising a plurality of wheels for operation over roads.

4. The computer vision system of claim 3, wherein the output subsystem is further coupled to a safety system in the vehicle for alerting a vehicle operator in response to receiving the motion detection signal.

5. A computer based method for detecting moving objects from a moving platform under low illumination conditions, the method comprising the steps of:

capturing image data of a scene from a moving platform at two or more points in time, the image data including low illumination imaging information and depth related information;

processing the image data to determine three-dimensional coordinates for points corresponding to a plurality of objects in the scene;

calculating an average scene disparity from changes in the three-dimensional coordinates for a first subset of objects in the scene likely to be stationary;

calculating an estimated position shift for each object in a second subset of objects in the scene based at least in part on the average scene disparity;

calculating an actual position shift for each object in the second subset of objects in the scene based, at least in part, on the three-dimensional coordinates for said second subset of objects;

calculating a position error for each object in the second subset of objects in the scene based, at least in part, on the difference between the estimated position shift and the actual position shift for each object; and

detecting at least one moving object from the objects in the second subset of objects based, at least in part, on a comparison between the position error for said at least one object and an empirically determined maximum threshold value.

6. The method of claim 5, wherein the depth related information comprises image data for a left frame and a right frame of the same scene.

7. The method of claim 5, wherein the image data comprises infrared-sensitive information for a first frame corresponding to the scene at a first point in time and for a second frame corresponding to the scene at a second point in time, and further wherein processing the image data comprises:

thresholding the image data to find one or more blobs of pixels in each frame corresponding to one or more heat emitting objects in the scene;

matching the blobs found in the first frame with the blobs found in the second frame corresponding to the same heat emitting objects; and

determining a three-dimensional coordinate set for each blob as a point in the blob.

8. The method of claim 7, wherein matching the blobs further comprises:

a first matching stage based on comparing blob at least blob area for each blob in a similar location within the first and the second frames; and

a second matching stage based on gray level template analysis, wherein gray level information within the image data for blobs in similar a location within the first and second frames are compared to determine their similarity.

9. The method of claim 5, wherein calculating the average scene disparity further comprises calculating a translational component based at least in part in a measured value provided by a speed measuring device.

10. The method of claim 5, wherein the maximum threshold value is based, at least in part, on a speed of the moving object and a distance of the moving object to the moving platform.

11. The method of claim 5, wherein detecting at least one moving object further comprises performing a pattern-matching analysis of the at least one object to determine whether the object matches a target object type.

12. The method of claim 11, wherein the target object type is human.

13. A computer system for detecting moving objects from a moving platform under low illumination conditions, the system comprising the steps of:

means for capturing image data of a scene from a moving platform at two or more points in time, the image data including low illumination imaging information and depth related information;

means for processing the image data to determine three-dimensional coordinates for points corresponding to a plurality of objects in the scene;

means for calculating an average scene disparity from changes in the three-dimensional coordinates for a first subset of objects in the scene likely to be stationary;

means for calculating an estimated position shift for each object in a second subset of objects in the scene based at least in part on the average scene disparity;

means for calculating an actual position shift for each object in the second subset of objects in the scene based, at least in part, on the three-dimensional coordinates for said second subset of objects;

means for calculating a position error for each object in the second subset of objects in the scene based, at least in part, on the difference between the estimated position shift and the actual position shift for each object; and

means for detecting at least one moving object from the objects in the second subset of objects based, at least in part, on a comparison between the position error for said at least one object and an empirically determined maximum threshold value.

14. The system of claim 13, wherein the depth related information comprises image data for a left frame and a right frame of the same scene.

15. The system of claim 13, wherein the image data comprises infrared-sensitive information for a first frame corresponding to the scene at a first point in time and for a second frame corresponding to the scene at a second point in time, and further wherein the means for processing the image data comprises:

means for thresholding the image data to find one or more blobs of pixels in each frame corresponding to one or more heat emitting objects in the scene;

means for matching the blobs found in the first frame with the blobs found in the second frame corresponding to the same heat emitting objects; and

means for determining a three-dimensional coordinate set for each blob as a point in the blob.

16. The system of claim 13, wherein the maximum threshold value is based, at least in part, on a speed of the moving object and a distance of the moving object to the moving platform.

17. A pedestrian detection system in a vehicle for assisting a driver to detect pedestrians and other moving objects in a path of the vehicle during low illumination conditions, the pedestrian detection system comprising:

an infrared stereo vision module for capturing scene image data at a plurality of times, the image data comprising information for deriving a depth measure for objects in the scene, the infrared stereo vision module comprising a plurality of infrared-sensitive cameras configured to operate under low illumination conditions;

a data processing module communicatively coupled to the infrared stereo vision module for receiving the image data and for providing motion detection data indicating the detection of independently moving objects within the scene, the data processing module configurable to process the image data to determine three-dimensional coordinates for points corresponding to a plurality of objects in the scene and further comprising;

an average scene disparity module configurable to calculate an average scene disparity from changes in three-dimensional coordinates for a first subset of objects in the scene likely to be stationary;

an estimated position shift module configurable to calculate an estimated position shift for each object in a second subset of objects in the scene based at least in part on the average scene disparity;

an actual position shift module configurable to calculate an actual position shift for each object in the second subset of objects in the scene based in part on the three-dimensional coordinates for said second subset of objects;

a position error module configurable to calculate a position error for each object in the second subset of objects in the scene based, at least in part, on the difference between the estimated position shift and the actual position shift for each object; and

an object detection module configurable to detect at least one moving object from the objects in the second subset of objects based, at least in part, on a comparison between the position error for said at least one object and an empirically determined maximum threshold value; and

an output interface coupled to the data processing module for receiving the motion detection data and for providing an output signal to a safety system processing unit within the vehicle for providing an alert signal to the driver in response to the detection of one or more pedestrians.

18. The pedestrian detection system of claim 17, wherein the safety system processing unit within the vehicle for providing an alert signal to the driver as one of an audible signal, a visually detectable signal, or a vehicle speed control signal.