Device and method for monitoring vehicle surroundings

• US 8,885,045 B2
• Filed: 07/31/2006
• Issued: 11/11/2014
• Est. Priority Date: 08/02/2005
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Yanagi, Takura
• Primary Examiner(s): Nguyen, Thanh Tammy
• Assistant Examiner(s): BUI, JONATHAN A
• Application Number:
  US11/997,515
• Publication Number:
  US 20100220189A1
• Time in Patent Office: 3,025 Days
• Field of Search: 348/147, 348/148, 348/152, 348/153
• US Class Current: 348/148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 19/483 with obstacle sensors of electric or electronic type
• CPC Class Codes: B60R 2300/102 using 360 degree surveillance camera system
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/70 characterised by an event-triggered choice to display a specific image among a selection of captured images
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60R 2300/8093 for obstacle warning

Abstract

A vehicle surrounding monitoring device include at least one camera 2 installed in an own vehicle to image a video around the own vehicle, an obstacle sensor 3 for detecting an obstacle within an imaging range of the camera 2, a pixel synthesis unit 13 for receiving a camera image imaged by the camera 2 and converting the camera image into a view point converted image seen from a virtual view point above the own vehicle, and a display device 4 for displaying the view point conversion image converted by the pixel synthesis unit 13. Simultaneously when a warning is given by a warning sound upon entry of an obstacle within an obstacle detection range of the obstacle sensor 3, an image of the obstacle detection range is synthesized with an obstacle image imaged by the camera 2 and included in the view point converted image to be in contact therewith.

Claims

1. A vehicle surrounding monitoring device, comprising:

at least one camera installed in an own vehicle to obtain an image around an own vehicle;

an obstacle detection device installed in the own vehicle configured to detect an obstacle in an obstacle detection range;

an image processing device configured to convert a camera image imaged by the camera into a view point converted image seen from a virtual view point above the own vehicle; and

a display configured to display the view point converted image of the image processing device, the view point converted image being synthesized with an image of the obstacle detection range of the obstacle detection device,

wherein, simultaneously when a warning is given upon entry of an obstacle within the obstacle detection range of the obstacle detection device, the image processing device modifies at least one of the image of the obstacle detection range of the obstacle detection device and an obstacle image imaged by the camera and included in the view point converted image so that the image of the obstacle detection range of the obstacle detection device and the obstacle image substantially contact one another,

wherein the image processing device is configured to set the image obtained by projecting the obstacle detection range of the obstacle detection device on a ground surface, with an attaching position of the camera set as a view point, as the image of the obstacle detection range, and is configured to modify the image of the obstacle detection range projected on the ground surface with the view point converted image seen from the virtual view point above the own vehicle.

2. The vehicle surrounding monitoring device according to claim 1, wherein the image processing device sterically modifies the image of the obstacle detection range of the obstacle detection device with the view point converted image.

3. The vehicle surrounding monitoring device according to claim 2, wherein the image of the obstacle detection range comprises a stereoscopic image that includes an image of an area where the obstacle detection range is projected on a ground surface, and a plane extending vertically from a boundary of an area where the obstacle detection range of the obstacle detection device is projected on the ground surface with an attaching position of the camera set as a view point, and is modified as the view point converted image to be displayed.

5. The vehicle surrounding monitoring device according to claim 1, wherein the image processing device is configured to display the image of the obstacle detection range in a semitransparent manner.

7. The vehicle surrounding monitoring device according to claim 1, wherein the warning is provided by a sound.

4. A vehicle surrounding monitoring device, comprising:

at least one camera installed in an own vehicle to obtain an image around an own vehicle;

an obstacle detection device installed in the own vehicle configured to detect an obstacle in an obstacle detection range;

an image processing device configured to convert a camera image imaged by the camera into a view point converted image seen from a virtual view point above the own vehicle; and

a display configured to display the view point converted image of the image processing device, the view point converted image being synthesized with an image of the obstacle detection range of the obstacle detection device,

wherein, simultaneously when a warning is given upon entry of an obstacle within the obstacle detection range of the obstacle detection device, the image processing device modifies at least one of the image of the obstacle detection range of the obstacle detection device and an obstacle image imaged by the camera and included in the view point converted image so that the image of the obstacle detection range of the obstacle detection device and the obstacle image substantially contact one another,

wherein the obstacle detection device is configured to detect an obstacle in space away from a ground surface; and

wherein an image of an area where the obstacle detection range is projected on the ground surface, and an image displayed in a position where the obstacle detected by the obstacle detection device is projected on the ground surface and brought into contact with the image of the area where the obstacle detection range is projected on the ground surface are modified as the view point converted image to be displayed,

wherein the image processing device is configured to set the image obtained by projecting the obstacle detection range of the obstacle detection device on a ground surface, with an attaching position of the camera set as a view point, as the image of the obstacle detection range, and is configured to modify the image of the obstacle detection range projected on the ground surface with the view point converted image seen from the virtual view point above the own vehicle.

6. A vehicle surrounding monitoring method, comprising:

obtaining an image around an own vehicle by at least one camera installed in the own vehicle;

detecting an obstacle in an obstacle detection range by an obstacle detection sensor;

converting the image obtained by the at least one camera into a view point converted image seen from a virtual view point above the own vehicle;

displaying the view point converted image synthesized with an image of the obstacle detection range of the obstacle detection sensor;

when presence of an obstacle is detected by the obstacle detection sensor, giving a warning, and simultaneously modifying at least one of the image of the obstacle detection range of the obstacle detection sensor and an obstacle image included in the view point converted image so that the image of the obstacle detection range of the obstacle detection sensor and the obstacle image substantially contact one another, setting an image obtained by projecting the obstacle detection range of the obstacle detection sensor on a ground surface, with an attaching position of the camera set as a view point, as the image of the obstacle detection range, and modifying the image of the obstacle detection range projected on the ground surface with the view point converted image seen from the virtual view point above the own vehicle.

8. A vehicle surrounding monitoring device, comprising:

at least one camera installed in an own vehicle to obtain an image around the own vehicle;

obstacle detection means installed in the own vehicle for detecting an obstacle in an obstacle detection range;

image processing means for converting a camera image imaged by the camera into a view point converted image seen from a virtual view point above the own vehicle; and

display means for displaying the view point converted image of the image processing means, the view point converted image being synthesized with an image of the obstacle detection range of the obstacle detection means,

wherein, simultaneously when a warning is given upon entry of an obstacle within the obstacle detection range of the obstacle detection means, the image processing means modifies at least one of the image of the obstacle detection range of the obstacle detection means and an obstacle image imaged by the camera and included in the view point converted image so that the image of the obstacle detection range of the obstacle detection means and the obstacle image contact one another,

wherein the image processing means is for setting an image obtained by projecting the obstacle detection range of the obstacle detection means on a ground surface, with an attaching position of the camera set as a view point, as the image of the obstacle detection range, and is for modifying the image of the obstacle detection range projected on the ground surface with the view point converted image seen from the virtual view point above the own vehicle.

9. A non-transitory computer readable medium having program code recorded therein, that, when executed on a computing system, causes the computing system to perform steps comprising:

obtaining an image around an own vehicle by at least one camera installed in the own vehicle;

detecting an obstacle in an obstacle detection range by an obstacle detection sensor;

converting the image obtained by the at least one camera into a view point converted image seen from a virtual view point above the own vehicle;

displaying the view point converted image synthesized with an image of the obstacle detection range of the obstacle detection sensor;

when presence of an obstacle is detected by the obstacle detection sensor, giving a warning, and simultaneously modifying at least one of the image of the obstacle detection range of the obstacle detection sensor and an obstacle image included in the view point converted image so that the image of the obstacle detection range of the obstacle detection sensor and the obstacle image substantially contact one another, setting an image obtained by projecting the obstacle detection range of the obstacle detection sensor on a ground surface, with an attaching position of the camera set as a view point, as the image of the obstacle detection range, and modifying the image of the obstacle detection range projected on the ground surface with the view point converted image seen from the virtual view point above the own vehicle.

10. A vehicle surrounding monitoring device, comprising:

at least one camera installed in an own vehicle to obtain an image around an own vehicle;

an obstacle detection device installed in the own vehicle configured to detect an obstacle in an obstacle detection range;

an image processing device configured to convert a camera image imaged by the camera into a view point converted image seen from a virtual view point above the own vehicle; and

a display configured to display the view point converted image of the image processing device, the view point converted image being synthesized with an image of the obstacle detection range of the obstacle detection device,

wherein, simultaneously when a warning is given upon entry of an obstacle within the obstacle detection range of the obstacle detection device, the image processing device modifies at least one of the image of the obstacle detection range of the obstacle detection device and an obstacle image imaged by the camera and included in the view point converted image so that the image of the obstacle detection range of the obstacle detection device and the obstacle image substantially contact one another,

wherein the display directly displays the view point converted image at a normal time when the obstacle detection device detects no obstacle, and displays the image of the obstacle detection range synthesized with the view point converted image when the obstacle detection device detects the obstacle,

wherein the image processing device is configured to set the image obtained by projecting the obstacle detection range of the obstacle detection device on a ground surface, with an attaching position of the camera set as a view point, as the image of the obstacle detection range, and is configured to modify the image of the obstacle detection range projected on the around surface with the view point converted image seen from the virtual view point above the own vehicle.