Monitoring device, monitoring method and program for monitoring

• US 7,266,220 B2
• Filed: 05/01/2003
• Issued: 09/04/2007
• Est. Priority Date: 05/09/2002
• Status: Active Grant

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Nobori, Kunio, Sato, Satoshi, Nakagawa, Masamichi
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US10/427,486
• Publication Number:
  US 20030210807A1
• Time in Patent Office: 1,587 Days
• Field of Search: 382/100, 382/103, 382/107, 382/104, 382/106, 340/933, 340/937, 348/113, 701/2, 701/41
• US Class Current: 382/104
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

In a monitoring device for vehicles, an optical flow detection section determines an optical flow from an image taken with a camera. A background flow estimation section determines a background flow, which is a flow for the camera image under the assumption that the image is a background, based on the motion of the vehicle. An approaching object detection section detects motion of an object in the surroundings of the vehicle by comparing the optical flow with the background flow.

Claims

1. A monitoring device comprising:

first means for determining an optical flow from an image taken with a camera capturing the surroundings of a vehicle;

second means for determining a background flow, which is an optical flow for the image under the assumption that the image is a background, based on motion of the vehicle and a space model modeling a space the camera is capturing, the space model identifying correspondence between points on the image and coordinates of a real world three-dimensional coordinate system; and

third means for detecting motion of an object existing in the surroundings of the vehicle by comparing the optical flow with the background flow.

2. The monitoring device of claim 1, wherein the space model is prepared based on distance data of objects the camera is capturing.

3. The monitoring device of claim 2, wherein the vehicle is provided with an obstacle sensor, and the distance data is measured with the obstacle sensor.

4. The monitoring device of claim 1, wherein the space model at least includes a road surface model modeling a road surface on which the vehicle is running.

5. The monitoring device of claim 1, wherein the space model at least includes a wall model assuming a wall normal to a road surface on which the vehicle is running.

6. The monitoring device of claim 5, wherein the wall is assumed to exist in the rear from the vehicle on a side.

7. The monitoring device of claim 1, wherein the third means determines whether or not the magnitude of the optical flow is larger than a predetermined value, and

if the magnitude is larger than the predetermined value, the comparison is made with use of an angle difference, or otherwise the comparison is made without use of an angle difference.

8. The monitoring device of claim 7, where in the predetermined value is set according to the magnitude of the background flow at a corresponding position on the image.

9. The monitoring device of claim 1, wherein the third means performs the steps of:

specifying approaching object candidate flows among optical flows by comparing the optical flows with background flows;

generating an approaching object candidate region by associating neighboring ones of the approaching object candidate flows with each other; and

determining the approaching object candidate flows related to the approaching object candidate region as noise when the area of the approaching object candidate region is smaller than a predetermined value.

10. The monitoring device of claim 1, wherein the second means determines as the background flow, a difference vector between camera coordinates at time t−

1 and at time t of a point on the space model, the camera coordinate at time t of the point being determined based on the motion of the vehicle from time t−

1 to time t.

11. A monitoring device comprising:

first means for determining an optical flow from an image taken with a camera capturing the surroundings of a vehicle,

second means for determining a space flow, which is motion of a point on the image in a real world coordinate system, based on the optical flow, motion of the vehicle and a space model modeling a space the camera is capturing, the space model identifying correspondence between points on the image and coordinates of a real world three-dimensional coordinate system; and

third means for detecting motion of an object existing in the surroundings of the vehicle based on the space flow.

12. A monitoring method comprising the steps of:

determining an optical flow from an image taken with a camera capturing the surroundings of a vehicle;

determining a background flow, which is an optical flow for the image under the assumption that the image is a background, based on motion of the vehicle and a space model modeling a space the camera is capturing, the space model identifying correspondence between points on the image and coordinates of a real world three-dimensional coordinate system; and

detecting motion of an object existing in the surroundings of the vehicle by comparing the optical flow with the background flow.

13. The monitoring method of claim 12, further comprising the step of estimating motion of the vehicle using a vehicle speed sensor and a steering angle sensor provided for the vehicle.

14. The monitoring method of claim 12, wherein a difference vector between camera coordinates at time t−

1 and at time t of a point on the space model is determined as the background flow, the camera coordinate at time t of the point being determined based on the motion of the vehicle from time t−

1 to time t.

15. A computer program embedded on a computer readable medium for monitoring, allowing a computer to execute the steps of:

determining an optical flow from an image taken with a camera capturing the surroundings of a vehicle;

determining a background flow, which is an optical flow for the image under the assumption that the image is a background, based on motion of the vehicle and a space model modeling a space the camera is capturing, the space model identifying correspondence between points on the image and coordinates of a real world three-dimensional coordinate system; and

detecting motion of an object existing in the surroundings of the vehicle by comparing the optical flow with the background flow.

16. The computer program of claim 15, wherein a difference vector between camera coordinates at time t−

1 and at time t of a point on the space model is determined as the background flow, the camera coordinate at time t of the point being determined based on the motion of the vehicle from time t−

1 to time t.