System for monitoring vehicle surroundings

• US 20050174429A1
• Filed: 02/03/2005
• Published: 08/11/2005
• Est. Priority Date: 02/04/2004
• Status: Abandoned Application

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Yanai, Tatsumi
• Application Number:
  US11/049,352
• Publication Number:
  US 20050174429A1
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 2001/1253 with cameras, video cameras or video screens
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60R 2300/8066 for monitoring rearward traffic

Abstract

An aspect of the present invention provides a vehicle surroundings monitoring system that includes, a plurality of cameras configured to photograph regions surrounding a vehicle, a surroundings monitoring control unit configured to determine areas of the images photographed by the cameras to be displayed based on the steering state of the vehicle or an input from a driver and to output image data that combines the images contained in the determined display areas in such a manner that items captured in the displayed images are arranged in positional relationships observed by a driver, and a display configured to display the image data outputted from the surroundings monitoring control unit, wherein the surroundings monitoring control unit is further configured such that it can control the proportion of the display that is occupied by each image displayed on the display.

Claims

1. A vehicle surroundings monitoring system, comprising:

a plurality of cameras configured to photograph regions surrounding a vehicle;

a surroundings monitoring control unit configured to determine areas of the images photographed by the cameras to be displayed based on the steering state of the vehicle or an input from a driver and to output image data that combines the images contained in the determined display areas in such a manner that items captured in the displayed images are arranged in positional relationships observed by a driver; and

a display configured to display the image data outputted from the surroundings monitoring control unit, wherein

the surroundings monitoring control unit is further configured such that it can control the proportion of the display that is occupied by each image displayed on the display.

2. The vehicle surroundings monitoring system as claimed in claim 1, wherein the surroundings monitoring control unit comprises:

a control unit configured to determine areas to be exsected from the camera images based on the steering state of the vehicle or an input from a driver and, based on the steering state of the vehicle or an input, determine an arrangement of the exsected images in positional relationships between items captured in the exsected images to be the positional relationships observed by a driver;

a display area setting unit configured to receive information from the control unit specifying the areas to be exsected from the images photographed by the cameras, exsect the specified areas from the images, and output image data describing the exsected images; and

an image combining unit configured to receive information specifying the arrangement of the exsected images and image data describing the exsected images from the control unit, arrange the exsected images in such a manner that the positional relationships between items captured in the exsected images is the positional relationships observed by a driver, and output image data describing the exsected images and arrangement of the exsected images to the display.

3. The vehicle surroundings monitoring system as claimed in claim 1, further comprising:

a steering state detecting unit configured to detect the steering state of the vehicle and output vehicle steering state information describing the steering state of the vehicle; and

an image selecting unit configured to select which of the photographed images to display on the display based on the vehicle steering state information and output image selection information indicating which images have been selected, wherein

the surroundings monitoring control unit comprises;

a control unit configured to determine areas to be exsected from the camera images based on the vehicle steering state information and the image selection information and, based on the vehicle steering state information, determine an arrangement of the exsected images that causes the positional relationships between items captured in the exsected images to be the positional relationships observed by a driver;

a display area setting unit configured to receive information from the control unit specifying the areas to be exsected from the images photographed by the cameras, exsect the specified areas from the images, and output image data describing the exsected images; and

an image combining unit configured to receive information specifying the arrangement of the exsected images and image data describing the exsected images from the control unit, arrange the exsected images in such a manner that the positional relationships between items captured in the exsected images are to be the positional relationships observed by a driver, and output image data to the display.

4. The vehicle surroundings monitoring system as claimed in claim 3, wherein

the steering state detecting unit is configured to detect if the gearshift of the vehicle is in a reverse gear position and detect the steering angle;

and

the control unit is configured to determine the areas to be exsected from the camera images based on the steering angle when the vehicle is in a reverse gear.

5. The vehicle surroundings monitoring system as claimed in claim 1, wherein the plurality of cameras comprises:

a first camera mounted to a rear section of the vehicle and is arranged and configured to photograph in a rearward direction;

a second camera mounted to a left side section of the vehicle and is arranged and configured to photograph a region located behind the left side section of the vehicle;

a third camera mounted to a right side section of the vehicle and is arranged and configured to photograph a region located behind the right side section of the vehicle.

6. The vehicle surroundings monitoring system as claimed in claim 3, wherein

the plurality of cameras comprises:

a first camera mounted to a rear section of the vehicle and is arranged and configured to photograph in a rearward direction,

a second camera mounted to a left side section of the vehicle and is arranged and configured to photograph a region located behind the left side section of the vehicle, and

a third camera mounted to a right side section of the vehicle and is arranged and configured to photograph a region located behind the right side section of the vehicle;

the steering state detecting unit is configured to detect if the gearshift of the vehicle is in a reverse gear position and detect the steering angle; and

the image selecting unit is configured to select the camera images of the first and second cameras so as to display an image showing regions behind the left side section of the vehicle and directly behind the vehicle when the vehicle is in a reverse gear and a leftward steering state and select the camera images of the first and third cameras so as to display an image showing regions behind the right side section of the vehicle and directly behind the vehicle when the vehicle is in a reverse gear and a rightward steering state.

7. The vehicle surroundings monitoring system as claimed in claim 6, wherein the control unit determines the display areas to be exsected from the images photographed with the cameras in such a manner that the following conditions are met:

when the steering angle is leftward, the larger the leftward steering angle is, the more the display area exsected from the camera image of the second camera is expanded outward in the leftward direction and the more the display area exsected from the camera image of the first camera is narrowed from the left and right toward the center of the image in the transverse direction of the vehicle;

when the steering angle is rightward, the larger the rightward steering angle is, the more the display area exsected from the camera image of the third camera is expanded outward in the rightward direction and the more the display area exsected from the camera image of the first camera is narrowed from the left and right toward the center of the image in the transverse direction of the vehicle;

when the steering angle is leftward, the smaller the leftward steering angle is, the more the display area exsected from the camera image of the second camera is narrowed with respect to the leftward outward direction and the more the display area exsected from the camera image of the first camera is expanded to the left and right away from the center of the image in the transverse direction of the vehicle; and

when the steering angle is rightward, the smaller the rightward steering angle is, the more the display area exsected from the camera image of the third camera is narrowed with respect to the rightward outward direction and the more the display area exsected from the camera image of the first camera is expanded to the left and right away from the center of the image in the transverse direction of the vehicle.

8. The vehicle surroundings monitoring system as claimed in claim 3, wherein

the plurality of cameras comprises:

a first camera mounted to a rear section of the vehicle and is arranged and configured to photograph in a rearward direction,

a second camera mounted to a left side section of the vehicle and is arranged and configured to photograph a region located behind the left side section of the vehicle, and

a third camera mounted to a right side section of the vehicle and is arranged and configured to photograph a region located behind the left side section of the vehicle;

the steering state detecting unit is configured to detect if the gearshift of the vehicle is in a reverse gear position and to detect the steering angle; and

the control unit is configured such that, when the vehicle is in reverse, it determines the display areas to be exsected from the images photographed with the cameras in such a manner that the following conditions are met;

the larger the leftward or rightward steering angle is, the more the display area exsected from the camera image of the second camera is expanded toward a leftward near region of the field of view, the more the display area exsected from the camera image of the third camera is expanded in the rightward near region of the field of view, and the more the display area exsected from the camera image of the first camera is narrowed to a region closer to the rear section of the vehicle, and

the smaller the leftward or rightward steering angle is, the more the display area exsected from the camera image of the second camera is narrowed to a leftward distant region of the field of view, the more the display area exsected from the camera image of the third camera is narrowed to a rightward distant region of the field of view, and the more the display area exsected from the camera image of the first camera is expanded toward a distant region of the field of view from the rear section of the vehicle.

9. The vehicle surroundings monitoring system as claimed in claim 2, further comprising:

a steering state detecting unit configured to detect the steering state of the vehicle and output vehicle steering state information describing the steering state of the vehicle;

an image count selector switch configured to receive input from a driver specifying the number of images to be displayed on the display;

an display area selector switch configured to receive input from a driver specifying the display areas to be displayed on the display; and

an image selecting unit configured to select camera images photographed by the cameras based on the steering state information, the input to the image count selection switch, and the input to the display area selector switch and output image selection information specifying which images were selected,

wherein the surroundings monitoring control unit comprises;

a control unit configured to determine areas to be exsected from the camera images based on the vehicle steering state information and the image selection information and, based on the vehicle steering state information, determine an arrangement of the exsected images in positional relationships between items captured in the exsected images to be the positional relationships observed by a driver;

a display area setting unit configured to receive information from the control unit specifying the areas to be exsected from the images photographed by the cameras, exsect the specified areas from the images, and output image data describing the exsected images; and

an image combining unit configured to receive information specifying the arrangement of the exsected images and image data describing the exsected images from the control unit, arrange the exsected images in such a manner that the positional relationships between items captured in the exsected images are arranged in positional relationships observed by a driver, and output image data describing the exsected images and the arrangement of the exsected images to the display.

10. The vehicle surroundings monitoring system as claimed in claim 2, further comprising:

an operation switch that can be operated by a driver; and

an advancement distance detecting unit configured to calculate the distance the vehicle advances after the operation switch is turned on,

wherein the plurality of cameras comprises;

a first camera mounted to a front section of the vehicle in a transversely central position with respect to the transverse direction of the vehicle and is arranged and configured to photograph in a forward and downward direction,

a second camera mounted to a front section of the vehicle and is arranged and configured to photograph in a direction leftward of the vehicle, and

a third camera mounted to a front section of the vehicle and is arranged and configured to photograph in a direction rightward of the vehicle; and

the control unit determines the display areas to be exsected from the images photographed with the cameras in such a manner that the following conditions are met;

when the advancement distance is small, the display area exsected from the camera image of the second camera is expanded from a leftward distant region of the field of view toward a nearer region of the field of view, the display area exsected from the camera image of the third camera is expanded from a rightward distant region of the field of view toward a nearer region of the field of view, and the display area exsected from the camera image of the first camera is narrowed to a region closer to the front section of the vehicle;

when the advancement distance is large, the display area exsected from the camera image of the second camera is narrowed to a leftward distant region of the field of view, the display area exsected from the camera image of the third camera is narrowed to a rightward distant region of the field of view, and the display area exsected from the camera image of the first camera is expanded toward a distant region of the field of view from the front section of the vehicle.

11. The vehicle surroundings monitoring system as claimed in claim 2, further comprising:

an operation switch that can be operated by a driver; and

an advancement distance detecting unit configured to calculate the distance the vehicle advances after the operation switch is turned on,

wherein the plurality of cameras comprises;

a first camera mounted to a rear section of the vehicle in a transversely central position with respect to the transverse direction of the vehicle and is arranged and configured to photograph in a rearward and downward direction,

a second camera mounted to a rear section of the vehicle and is arranged and configured to photograph in a direction leftward of the vehicle, and

a third camera mounted to a rear section of the vehicle and is arranged and configured to photograph in a direction rightward of the vehicle;

the control unit determines the display areas to be exsected from the images photographed with the cameras in such a manner that the following conditions are met;

when the advancement distance is small, the display area exsected from the camera image of the second camera is expanded from a leftward distant region of the field of view toward a nearer region of the field of view, the display area exsected from the camera image of the third camera is expanded from a rightward distant region of the field of view toward a nearer region of the field of view, and the display area exsected from the camera image of the first camera is narrowed to a region closer to the rear section of the vehicle;

when the advancement distance is large, the display area exsected from the camera image of the second camera is narrowed to a leftward distant region of the field of view, the display area exsected from the camera image of the third camera is narrowed to a rightward distant region of the field of view, and the display area exsected from the camera image of the first camera is expanded toward a distant region of the field of view from the rear section of the vehicle.

12. The vehicle surroundings monitoring system as claimed in claim 2, further comprising:

a feature extracting unit configured to extract feature alignment points existing on the ground in the images exsected by the display area setting unit,

wherein the image combining unit arranges the exsected images in such a manner that the feature alignment points extracted from the display areas of the camera images by the feature extracting unit are drawn closer together.

13. The vehicle surroundings monitoring system as claimed in claim 6, further comprising:

a photographing direction setting unit configured to set the photographing directions of the second camera and the third camera based on the steering angle detected steering state detecting unit; and

actuators configured to change the photographing directions of the second camera and third camera based on the photographing directions of the second camera and third camera set by the photographing direction setting unit.

14. The vehicle surroundings monitoring system as claimed in claim 9, further comprising:

a photographing direction setting unit configured to set the photographing directions of the second camera and the third camera based on the input to the display area selector switch; and

an actuator(s) configured to change the photographing directions of the second camera and third camera based on the photographing directions of the second camera and third camera set by the photographing direction setting unit.

15. A vehicle with a surroundings monitoring system, comprising:

a plurality of cameras configured to photograph regions surrounding a vehicle;

a surroundings monitoring control unit configured to determine areas of the images photographed by the cameras to be displayed based on the steering state of the vehicle or an input from a driver and to output image data that combines the images contained in the determined display areas in such a manner that items captured in the displayed images are arranged in positional relationships observed by a driver; and

a display configured to display the image data outputted from the surroundings monitoring control unit, wherein

the surroundings monitoring control unit is further configured such that it can control the proportion of the display that is occupied by each image displayed on the display.

16. The vehicle as claimed in claim 15, wherein the surroundings monitoring control unit comprises:

a control unit configured to determine areas to be exsected from the camera images based on the steering state of the vehicle or an input from a driver and, based on the steering state of the vehicle or an input, determine an arrangement of the exsected images in positional relationships between items captured in the exsected images to be the positional relationships observed by a driver;

a display area setting unit configured to receive information from the control unit specifying the areas to be exsected from the images photographed by the cameras, exsect the specified areas from the images, and output image data describing the exsected images; and

an image combining unit configured to receive information specifying the arrangement of the exsected images and image data describing the exsected images from the control unit, arrange the exsected images in such a manner that the positional relationships between items captured in the exsected images is the positional relationships observed by a driver, and output image data describing the exsected images and arrangement of the exsected images to the display.

17. A method of monitoring the surroundings of a vehicle, comprising:

photographing a plurality of images of regions surrounding the vehicle;

determining areas of the photographed images to be displayed on a display based on the steering state of the vehicle or an input from a driver; and

combining the images contained in the determined display areas to display in such a manner that the positional relationships between items captured in the displayed images are to be the positional relationships observed by a driver;

wherein the proportion of the display occupied by each image displayed on the display is controlled.

18. The vehicle surroundings monitoring method as claimed in claim 17, wherein

when the areas of the photographed images to be displayed on the display are determined based on the steering state of the vehicle or an input from the driver, the areas to be exsected from the photographed images are determined based on the steering state of the vehicle or an input from the driver;

and

when the images contained in the determined display areas are combined and displayed in such a manner that the positional relationships between items captured in the displayed images are to be the positional relationships observed by a driver, an arrangement of the images exsected based on the steering state of the vehicle or an input from the driver is determined in which the positional relationships between items captured in the displayed images are to be the positional relationships observed by a driver and the exsected display areas are displayed in the determined arrangement.

19. The vehicle surroundings monitoring method as claimed in claim 18, further comprising:

detecting the steering state of the vehicle;

selecting photographed images to display based on the detected vehicle steering state;

wherein when the areas of the photographed images to be displayed on the display are determined based on the steering state of the vehicle or an input from the driver, the areas to be exsected from the photographed images are combined based on the detected steering state; and

when the images contained in the determined display areas are combined and to display in such a manner that the positional relationships between items captured in the displayed images are to be the positional relationships observed by a driver, the determined display areas are exsected from the selected images, an arrangement of the exsected display areas is determined in which the positional relationships between items captured in the displayed images are to be the positional relationships observed by a driver, and the exsected display areas are displayed in the determined arrangement.

20. The vehicle surroundings monitoring method as claimed in claim 18, further comprising:

detecting the steering state of the vehicle;

receiving input specifying the number of images to be displayed from the driver;

receiving input specifying the image display areas to be displayed from the driver; and

selecting images from among the plurality of photographed images based on the detected steering state, the input specifying the number of images to be displayed, and the input specifying the image display areas to be displayed,

wherein when the areas of the photographed images to be displayed on the display are determined based on the steering state of the vehicle or an input from the driver, the areas to be exsected from the photographed images are determined based on the detected steering state, the input specifying the number of images to be displayed, and the input specifying the image display areas to be displayed;

when the images contained in the determined display areas are combined and displayed in such a manner that the positional relationships between items captured in the displayed images are to be the positional relationships observed by a driver, an arrangement of the display areas in which the positional relationships between items captured in the displayed images are to be the positional relationships observed by a driver is determined based on the detected steering state, the determined display areas are exsected from the selected images, and the exsected display areas are displayed in the determined arrangement.