Surroundings monitoring apparatus

• US 10,632,915 B2
• Filed: 09/14/2016
• Issued: 04/28/2020
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Aisin Seiki Co., Ltd. (Aisin Corp.)
• Original Assignee: Aisin Seiki Co., Ltd. (Aisin Corp.)
• Inventors: Yamamoto, Kinji, Kubota, Naotaka, Watanabe, Kazuya, Nakasho, Takayuki
• Primary Examiner(s): Dang, Philip P.
• Application Number:
  US15/755,285
• Publication Number:
  US 20180229657A1
• Time in Patent Office: 1,322 Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 11/02 for radio sets, television sets, telephones, or the like; Arrangement of controls thereof
• CPC Class Codes: B60R 2300/102 using 360 degree surveillance camera system
• CPC Class Codes: B60R 2300/20 characterised by the type of display used
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/308 by overlaying the real scene, e.g. through a head-up display on the windscreen
• CPC Class Codes: B60R 2300/605 the adjustment being automatic
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/8066 for monitoring rearward traffic
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A surroundings monitoring apparatus includes an image acquisition portion acquiring captured image data, an image conversion portion converting the captured image data to virtual image data with a plurality of virtual viewpoints, and a data control portion sequentially displaying the virtual image data at a display unit in a state where each of the virtual viewpoints moves from a first virtual viewpoint position at which an opposite side of the vehicle is viewed, through a second virtual viewpoint position at which the vehicle is viewed from an overhead region of the vehicle, and to a third virtual viewpoint position at which the one side of the vehicle is viewed. The data control portion rotates the virtual image data with reference to a line of sight of the virtual viewpoint while the virtual image data is displayed so that the virtual viewpoint passes through the second virtual viewpoint position.

Claims

1. A surroundings monitoring apparatus comprising:

a processor configured to

acquire captured image data output from an imaging portion which is mounted at a vehicle to capture an image of surroundings of the vehicle;

convert the captured image data to virtual image data with a plurality of virtual viewpoints serving as imaging directions towards the vehicle; and

sequentially display the virtual image data at a display unit provided at a vehicle interior of the vehicle in a state where each of the virtual viewpoints moves from a first virtual viewpoint position at which an opposite side of the vehicle is viewed beyond a whole of the vehicle from one side of the vehicle, through a second virtual viewpoint position at which the vehicle is viewed from an overhead region of the vehicle, and to a third virtual viewpoint position at which the one side of the vehicle is viewed beyond the whole of the vehicle from the other side of the vehicle, wherein

the whole of the vehicle is displayed together with the virtual image data during a period when the virtual viewpoint moves from the first virtual viewpoint position, through the second virtual viewpoint position, and to the third virtual viewpoint position, and

the virtual image data is rotated with reference to a line of sight of the virtual viewpoint while the virtual image data is displayed so that the virtual viewpoint passes through the second virtual viewpoint position,

wherein the second virtual viewpoint position is positioned directly above the vehicle, and

wherein the control portion rotates the virtual image data is rotated by 180°

at the second virtual viewpoint position which is positioned directly above the vehicle.

2. The surroundings monitoring apparatus according to claim 1, wherein the third virtual viewpoint position is positioned at a rear side relative to an expected travelling direction of the vehicle in a case where the vehicle is started.

3. The surroundings monitoring apparatus according to claim 1, wherein the image conversion portion converts the captured image data to the virtual image data by projecting the captured image data on a curved surface with reference to the position of the vehicle.

4. The surroundings monitoring apparatus according to claim 1, wherein the captured image data is converted to the virtual image data by projecting the captured image data on a plane with reference to the position of the vehicle.

5. The surroundings monitoring apparatus according to claim 1, wherein an image of an own vehicle is displayed by superimposing the image on the virtual image data.

6. The surroundings monitoring apparatus according to claim 1, wherein

the virtual image data is displayed together with the vehicle so that the vehicle is between the virtual viewpoint and the virtual image data and positioned at or near center of a display area of the display unit while the virtual viewpoint moves from the first virtual viewpoint position, through the second virtual viewpoint position, and to the third virtual viewpoint position.

7. A surroundings monitoring method comprising:

acquiring captured image data output from an imaging portion which is mounted at a vehicle to capture an image of surroundings of the vehicle;

converting the captured image data to virtual image data with a plurality of virtual viewpoints serving as imaging directions towards the vehicle; and

sequentially displaying the virtual image data at a display unit provided at a vehicle interior of the vehicle in a state where each of the virtual viewpoints moves from a first virtual viewpoint position at which an opposite side of the vehicle is viewed beyond a whole of the vehicle from one side of the vehicle, through a second virtual viewpoint position at which the vehicle is viewed from an overhead region of the vehicle, and to a third virtual viewpoint position at which the one side of the vehicle is viewed beyond the whole of the vehicle from the other side of the vehicle, wherein

the whole of the vehicle is displayed together with the virtual image data during a period when the virtual viewpoint moves from the first virtual viewpoint position, through the second virtual viewpoint position, and to the third virtual viewpoint position, and

the virtual image data is rotated with reference to a line of sight of the virtual viewpoint while the virtual image data is displayed so that the virtual viewpoint passes through the second virtual viewpoint position,

wherein the second virtual viewpoint position is positioned directly above the vehicle, and

wherein the control portion rotates the virtual image data is rotated by 180°

at the second virtual viewpoint position which is positioned directly above the vehicle.

8. The surroundings monitoring method according to claim 7, wherein the third virtual viewpoint position is positioned at a rear side relative to an expected travelling direction of the vehicle in a case where the vehicle is started.

9. The surroundings monitoring method according to claim 7, wherein the image conversion portion converts the captured image data to the virtual image data by projecting the captured image data on a curved surface with reference to the position of the vehicle.

10. The surroundings monitoring method according to claim 7, wherein the captured image data is converted to the virtual image data by projecting the captured image data on a plane with reference to the position of the vehicle.

11. The surroundings monitoring method according to claim 7, wherein an image of an own vehicle is displayed by superimposing the image on the virtual image data.

12. The surroundings monitoring method according to claim 7, wherein the virtual image data is displayed together with the vehicle so that the vehicle is between the virtual viewpoint and the virtual image data and positioned at or near center of a display area of the display unit while the virtual viewpoint moves from the first virtual viewpoint position, through the second virtual viewpoint position, and to the third virtual viewpoint position.