Moving object recognition systems, moving object recognition programs, and moving object recognition methods

• US 9,824,586 B2
• Filed: 11/29/2012
• Issued: 11/21/2017
• Est. Priority Date: 07/31/2012
• Status: Active Grant

• Current Assignee: DENSO Corporation, Denso IT Laboratory, Inc. (DENSO Corporation)
• Original Assignee: DENSO Corporation, Denso IT Laboratory, Inc. (DENSO Corporation)
• Inventors: Sato, Ikuro, Tamatsu, Yukimasa, Goto, Kunihiro
• Primary Examiner(s): FUJITA, KATRINA R
• Application Number:
  US13/689,196
• Publication Number:
  US 20140037138A1
• Time in Patent Office: 1,818 Days
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

The moving object recognition system includes: a camera that is installed in a vehicle and captures continuous single-view images; a moving object detecting unit that detects a moving object from the images captured by the camera; a relative approach angle estimating unit that estimates the relative approach angle of the moving object detected by the moving object detecting unit with respect to the camera; a collision risk calculating unit that calculates the risk of the moving object colliding with the vehicle, based on the relationship between the relative approach angle and the moving object direction from the camera toward the moving object; and a reporting unit that reports a danger to the driver of the vehicle in accordance with the risk calculated by the collision risk calculating unit.

Claims

1. A moving object recognition system comprising:

a camera configured to capture a plurality of continuous single-view images, the camera being installed in a vehicle;

a moving object detecting unit configured to detect one or more moving objects from the images captured by the camera;

a relative approach angle estimating unit configured to estimate a relative approach angle of a moving object detected by the moving object detecting unit with respect to the camera and translating with the vehicle, among the one or more moving objects, wherein the relative approach angle is an angle between a vector of translational motion of the moving object with respect to the vehicle and a vector of translational motion of a stationary object with respect to the vehicle; and

a collision risk calculating unit configured to calculate a risk of the moving object colliding with the vehicle, based on a relationship between the relative approach angle and a moving object direction from the camera toward the moving object, wherein the collision risk calculating unit makes the risk higher as a difference between the relative approach angle and the moving object direction becomes smaller.

2. The moving object recognition system of claim 1, wherein the collision risk calculating unit sets a danger zone based on the relative approach angle, and calculates the risk based on a probability of the moving object existing in the danger zone.

3. The moving object recognition system of claim 2, wherein the danger zone is set in an X-Z plane where the camera is the point of origin, an optical axis direction of the camera is a Z direction, a vertical direction is a Y direction, and a horizontal direction is an X direction, the danger zone being a region extending from the vehicle in a direction at the relative approach angle and having a width of the vehicle.

4. The moving object recognition system of claim 3, wherein the collision risk calculating unit includes:

a distance estimating unit configured to estimate an upper limit value of a distance from the camera to the moving object in the Z direction; and

a risk calculating unit configured to calculate the risk by determining a proportion of a line segment existing in the danger zone to a line segment of an XZ ray between the camera and the upper limit value, the XZ ray being a line segment fanned by orthographically projecting a straight line onto the X-Z plane, the straight line passing through the camera and the moving object in the images located at a focal distance from the camera.

5. The moving object recognition system of claim 4, wherein

the moving object detecting unit includes:

an optical flow calculating unit configured to extract a plurality of feature points from the images, and calculate optical flows of the extracted feature points; and

a grouping unit configured to group, among the extracted feature points, a plurality of feature points having the optical flows extended therefrom and converging at a vanishing point as a plurality of feature points existing on the moving object, and

the distance estimating unit selects a lowest feature point having the lowest height among the grouped feature points existing on the moving object, and estimates the upper limit value of the distance from the camera to the moving object in the Z direction by calculating a Z-coordinate of a ground surface point that is an intersection point between a straight line connecting the lowest feature point to the optical center and a ground surface on which the vehicle is running.

6. The moving object recognition system of claim 3, wherein the collision risk calculating unit includes:

a distance estimating unit configured to estimate a distance from the camera to the moving object in the Z direction; and

a risk calculating unit configured to make the risk higher when a point at the estimated distance from the camera on an XZ ray exists in the danger zone than when the point does not exist in the danger zone, the XZ ray being a line segment formed by orthographically projecting a straight line onto the X-Z plane, the straight line passing through the camera and the moving object in the images located at a focal distance from the camera.

7. The moving object recognition system of claim 6, wherein

the moving object detecting unit includes:

an optical flow calculating unit configured to extract a plurality of feature points from the images, and calculate optical flows of the extracted feature points; and

a grouping unit configured to group, among the extracted feature points, a plurality of feature points having the optical flows extended therefrom and converging at a vanishing point as a plurality of feature points existing on the moving object, and

the distance estimating unit calculates, based on the vanishing point, a time to collision of the moving object with the vehicle, and estimates, based on the time to collision and a velocity of the vehicle, the distance from the camera to the moving object in the Z direction.

8. The moving object recognition system of claim 3, wherein the collision risk calculating unit includes:

a distance estimating unit configured to determine a probability distribution of a distance from the camera to the moving object in the Z direction; and

a risk calculating unit configured to calculate the risk, based on a value of the integral of a probability density function of the probability distribution within a distance range in which an XZ ray exists in the danger zone, the XZ ray being a line segment formed by orthographically projecting a straight line onto the X-Z plane, the straight line passing through the camera and the moving object in the images located at a focal distance from the camera.

9. The moving object recognition system of claim 8, wherein

the moving object detecting unit includes:

an optical flow calculating unit configured to extract a plurality of feature points from the images, and calculate optical flows of the extracted feature points; and

a grouping unit configured to group, among the extracted feature points, a plurality of feature points having the optical flows extended therefrom and converging at a vanishing point as a plurality of feature points existing on the moving object, and

the distance estimating unit calculates, based on the vanishing point, a time to collision of the moving object with the vehicle, and determines, based on the time to collision, the probability distribution of the distance in compliance with a predetermined normal distribution.

10. The moving object recognition system of claim 3, wherein the collision risk calculating unit includes:

a distance estimating unit configured to estimate an upper limit value of a distance from the camera to the moving object in the Z direction, and determine a probability distribution of the distance from the camera to the moving object in the Z direction; and

a risk calculating unit configured to calculate the risk, based on a proportion of a value of the integral of a probability density function of the probability distribution within a distance range in which an XZ ray exists in the danger zone to a value of the integral of a probability density function of the probability distribution within a distance range up to the upper limit value, the XZ ray being a line segment formed by orthographically projecting a straight line onto the X-Z plane, the straight line passing through the camera and the moving object in the images located at a focal distance from the camera.

11. The moving object recognition system of claim 10, wherein

the moving object detecting unit includes:

an optical flow calculating unit configured to extract a plurality of feature points from the linages, and calculate optical flows of the extracted feature points; and

a grouping unit configured to group, among the extracted feature points, a plurality of feature points having the optical flows extended therefrom and converging at a vanishing point as a plurality of feature points existing on the moving object, and

the distance estimating unit selects a lowest feature point having the lowest height among the grouped feature points existing on the moving object, estimates the upper limit value of the distance from the camera to the moving object in the Z direction by calculating a Z-coordinate of a ground surface point that is an intersection point between a straight line connecting the lowest feature point to the optical center and a ground surface on which the vehicle is running, calculates, based on the vanishing point, a time to collision of the moving object with the vehicle, and determines, based on the time to collision, the probability distribution of the distance in compliance with a predetermined normal distribution.

12. The moving object recognition system of claim 2, wherein the collision risk calculating unit divides the danger zone into a plurality of levels, and sets the levels in a stepwise fashion.

13. The moving object recognition system of claim 1, further comprising a reporting unit configured to report a danger to a driver of the vehicle in accordance with the risk calculated by the collision risk calculating unit.

14. The moving object recognition system of claim 1, wherein the collision risk calculating unit corrects the risk, based on a time to collision of the moving object with the vehicle.

15. A moving object recognition system comprising:

a camera configured to capture a plurality of continuous single-view images, the camera being installed in a vehicle;

a moving object detecting unit configured to detect one or more moving objects by extracting a plurality of feature points from the images captured by the camera, generating optical flows of the extracted feature points, grouping, among the extracted feature points, a plurality of feature points having the optical flows extended therefrom and converging at a vanishing point, and detecting the grouped feature points as the moving object; and

a collision risk calculating unit configured to calculate a risk of a moving object detected by the moving object detecting unit, translating with the vehicle and colliding with the vehicle, among the one or more moving objects,

wherein the collision risk calculating unit calculates the risk of the moving object in such a manner that the risk of the moving object located at a short distance from the vanishing point in the images becomes higher than the risk of the moving object at a long distance from the vanishing point in the images, and

wherein the collision risk calculating unit makes the risk higher as a difference between a relative approach angle and a moving object direction becomes smaller, the relative approach angle being an angle between a vector of translational motion of the moving object with respect to the vehicle and a vector of translational motion of a stationary object with respect to the vehicle.

16. The moving object recognition system of claim 15, wherein the collision risk calculating unit calculates a risk of a representative point with the vehicle, the representative point being one of the feature points existing on the moving object or a point determined from the feature points existing on the moving object.

17. A non-transitory computer-readable medium having stored therein a moving object recognition program causing a computer to function as:

a moving object detecting unit configured to detect one or more moving objects from a plurality of continuous single-view images captured by a camera installed in a vehicle;

a relative approach angle estimating unit configured to estimate a relative approach angle of a moving object detected by the moving object detecting unit with respect to the camera and translating with the vehicle, among the one or more moving objects, wherein the relative approach angle is an angle between a vector of translational motion of the moving object with respect to the vehicle and a vector of translational motion of a stationary object with respect to the vehicle; and

a collision risk calculating unit configured to calculate a risk of the moving object colliding with the vehicle, based on a relationship between the relative approach angle and a moving object direction from the camera toward the moving object, wherein the collision risk calculating unit makes the risk higher as a difference between the relative approach angle and the moving object direction becomes smaller.

18. A non-transitory computer-readable medium having stored therein a moving object recognition program causing a computer to function as:

a moving object detecting unit configured to detect one or more moving objects by extracting a plurality of feature points from a plurality of continuous single-view images captured by a camera installed in a vehicle, generating optical flows of the extracted feature points, grouping, among the extracted feature points, a plurality of feature points having the optical flows extended therefrom and converging at a vanishing point, and detecting the grouped feature points as the moving object; and

a collision risk calculating unit configured to calculate a risk of a moving object detected by the moving object detecting unit, translating with the vehicle and colliding with the vehicle, among the one or more moving objects, in such a manner that the risk of the moving object located at a short distance from the vanishing point in the images becomes higher than the risk of the moving object at a long distance from the vanishing point in the images,

wherein the collision risk calculating unit makes the risk higher as a difference between a relative approach angle and a moving object direction becomes smaller, the relative approach angle being an angle between a vector of translational motion of the moving object with respect to the vehicle and a vector of translational motion of a stationary object with respect to the vehicle.

19. A moving object recognition method comprising:

capturing a plurality of continuous single-view images with a camera installed in a vehicle;

detecting one or more moving objects from the images captured by the camera;

estimating a relative approach angle of a moving object detected in the detecting step with respect to the camera and translating with the vehicle, among the one or more moving objects, wherein the relative approach angle is an angle between a vector of translational motion of the moving object with respect to the vehicle and a vector of translational motion of a stationary object with respect to the vehicle; and

calculating a risk of the moving object colliding with the vehicle based on a relationship between the relative approach angle and a moving object direction from the camera toward the moving object, wherein the collision risk calculating step makes the risk higher as a difference between the relative approach angle and the moving object direction becomes smaller.

20. A moving object recognition method comprising:

capturing a plurality of continuous single-view images with a camera installed in a vehicle;

detecting one or more moving objects by extracting a plurality of feature points from the images captured by the camera, generating optical flows of the extracted feature points, grouping, among the extracted feature points, a plurality of feature points having the optical flows extended therefrom and converging at a vanishing point, and detecting the grouped feature points as the moving object; and

calculating a risk of a moving object detected in the detecting step, translating with the vehicle and colliding with the vehicle, among the one or more moving objects, and

wherein the risk is calculated in such a manner that the risk of the moving object located at a short distance from the vanishing point in the images becomes higher than the risk of the moving object at a long distance from the vanishing point in the images, and

wherein the collision risk calculating step makes the risk higher as a difference between a relative approach angle and a moving object direction becomes smaller, the relative approach angle being an angle between a vector of translational motion of the moving object with respect to the vehicle and a vector of translational motion of a stationary object with respect to the vehicle.