In-vehicle image processing method and image processing apparatus

• US 8,576,285 B2
• Filed: 12/29/2009
• Issued: 11/05/2013
• Est. Priority Date: 03/25/2009
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Gomi, Toshiaki, Kawai, Jun, Yano, Katsutoshi, Yamada, Hiroshi
• Primary Examiner(s): Rao, Andy
• Application Number:
  US12/654,673
• Publication Number:
  US 20100245573A1
• Time in Patent Office: 1,407 Days
• Field of Search: 348/113, 348/116, 348/118, 348/122, 348/148, 382/154, 382/123, 382/101, 382/285, 382/291
• US Class Current: 348/148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/102 using 360 degree surveillance camera system
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/602 with an adjustable viewpoint
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An image processing apparatus is provided that combines image data captured by multiple cameras and displays the resulting image of surroundings of the vehicle to monitor monitoring the surroundings of a vehicle. The image processing apparatus includes: an image memory; a camera-image acquisition unit; a projection surface geometry manager that manages projection surface geometry information including a set of three-dimensional coordinates for a projection surface that is virtually formed to surround the vehicle; a viewing movement controller that determines viewing information so that a viewpoint and a viewing direction relative to the projection surface are continuously transited according to a given viewing transition regulation; an image converter that projects and combines the image data on the projection surface, and converts the combined image data into two-dimensional image data; a display unit that displays the two-dimensional image data; and a periodic rendering trigger generator that generates a trigger signal to start an image conversion process by the image converter.

Claims

1. An in-vehicle image processing apparatus capable of being linked to multiple cameras mounted to a vehicle, the image processing apparatus comprising:

an image memory that stores multiple image data captured by each of the multiple cameras;

a camera-image acquisition unit that inputs and stores the multiple image data captured by each of the cameras into the image memory;

a projection surface geometry manager that manages projection surface geometry information including a set of three-dimensional coordinates for a projection surface that is virtually formed to surround the vehicle;

a viewing movement controller that determines viewing information so that a viewpoint and a viewing direction relative to the projection surface are continuously transited according to given viewing transition regulations;

an image converter that projects and combines each of the image data stored in the image memory on the projection surface, and converts the combined image data into two-dimensional image data based on the viewing information determined by the viewing movement controller;

a display unit that displays the two-dimensional image data converted by the image converter;

a periodic rendering trigger generator that generates a trigger signal to make the image converter start an image conversion process;

a vehicle-profile memory that stores the three-dimensional profile information of the vehicle; and

a minimum elevation angle memory that determines and stores a minimum elevation angle for viewing so that a range of the image data shielded by the three-dimensional profile information of the vehicle has a given distance or less, when the three-dimensional profile information of the vehicle is combined with each of the image data projected on the projection surface, wherein

the image converter reads the three-dimensional profile information of the vehicle from the vehicle-profile memory, combines the three-dimensional profile information of the vehicle with each of the image data projected to the projection surface, and converts the combined image data into two-dimensional image data, and

the viewing movement controller corrects the viewing information based on the minimum elevation angle for viewing stored in the minimum elevation angle memory.

2. An in-vehicle image processing apparatus capable of being linked to multiple cameras mounted to a vehicle, the image processing apparatus comprising:

an image memory that stores multiple image data captured by each of the multiple cameras;

a camera-image acquisition unit that inputs and stores the multiple image data captured by each of the cameras into the image memory;

a projection surface geometry manager that manages projection surface geometry information including a set of three-dimensional coordinates for a projection surface that is virtually formed to surround the vehicle;

a viewing movement controller that determines viewing information so that a viewpoint and a viewing direction relative to the projection surface are continuously transited according to given viewing transition regulations;

an image converter that projects and combines each of the image data stored in the image memory on the projection surface, and converts the combined image data into two-dimensional image data based on the viewing information determined by the viewing movement controller;

a display unit that displays the two-dimensional image data converted by the image converter;

a periodic rendering trigger generator that generates a trigger signal to make the image converter start an image conversion process; and

a viewing transition-regulation memory that stores the viewing transition regulations, wherein

the viewing transition regulations stored in the viewing transition-regulation memory include start-point viewing information, end-point viewing information and viewing interpolation regulations,

the viewing movement controller continuously changes the viewing information based on the start-point viewing information, the end-point viewing information, and the viewing interpolation regulations in the viewing transition-regulation memory, and

the viewing interpolation regulations stored in the viewing transition-regulation memory continuously increases a transition speed for a given period of time after the transition of viewing information is started from the start-point viewing information, and continuously decreases the transition speed for a given period of time before the transition of viewing information is stopped at the end-point viewing information.

3. The image processing apparatus according to claim 2, wherein

the vehicle is further provided with an obstacle sensor that is capable to link to the image processing apparatus and detects obstacle information including at least a position an obstacle exists, and

the viewing movement controller receives the obstacle information, determines whether an obstacle is present on two-dimensional image data to be converted or not, and reduces the interpolation width in the viewing interpolation regulation to decrease the transition speed of viewing information when determining that an obstacle is present on the two-dimensional image data.

4. The image processing apparatus according to claim 2, further comprising:

a field-of-view-range detector that detects a range of two-dimensional image data converted by the image converter that is included in the projection surface, wherein the field-of-view-range detector determines whether the two-dimensional image data converted by the image converter covers the overall directions at the time when the viewing information transited by the viewing movement controller reaches the end-point viewing information, and when determining that the overall directions are not covered, changes the end-point viewing information.

5. An image processing method for in-vehicle image processing apparatus capable of being linked to multiple cameras mounted to a vehicle, the method comprising:

storing multiple image data captured by each of the multiple cameras;

inputting and storing the multiple image data captured by each of the cameras into the image memory;

managing projection surface geometry information including a set of three-dimensional coordinates for a projection surface that is virtually formed to surround the vehicle;

determining viewing information so that a viewpoint and a viewing direction relative to the projection surface are continuously transited according to given viewing transition regulations;

projecting and combining each of the stored image data on the projection surface, and converting the combined image data into two-dimensional image data based on the determined viewing information;

displaying the converted two-dimensional image data;

generating a trigger signal to make the converting start an image conversion process;

storing the three-dimensional profile information of the vehicle; and

determining and storing a minimum elevation angle for viewing so that a range of the image data shielded by the three-dimensional profile information of the vehicle has a given distance or less, when the three-dimensional profile information of the vehicle is combined with each of the image data projected on the projection surface, wherein

the converting includes reading the stored three-dimensional profile information of the vehicle, combining the three-dimensional profile information of the vehicle with each of the image data projected to the projection surface, and converting the combined image data into two-dimensional image data, and

the determining viewing information includes correcting the viewing information based on the stored minimum elevation angle for viewing.

6. An image processing method for in-vehicle image processing apparatus capable of being linked to multiple cameras mounted to a vehicle, the method comprising:

storing multiple image data captured by each of the multiple cameras;

inputting and storing the multiple image data captured by each of the cameras into the image memory;

managing projection surface geometry information including a set of three-dimensional coordinates for a projection surface that is virtually formed to surround the vehicle;

determining viewing information so that a viewpoint and a viewing direction relative to the projection surface are continuously transited according to given viewing transition regulations;

projecting and combining each of the stored image data on the projection surface, and converting the combined image data into two-dimensional image data based on the determined viewing information;

displaying the converted two-dimensional image data;

generating a trigger signal to make the converting start an image conversion process; and

storing the viewing transition regulations, wherein

the stored viewing transition regulations include start-point viewing information, end-point viewing information and viewing interpolation regulations,

the determining viewing information includes continuously changing the viewing information based on the start-point viewing information, the end-point viewing information, and the viewing interpolation regulations, and

the stored viewing interpolation regulations stored in the viewing transition-regulation memory continuously increase a transition speed for a given period of time after the transition of viewing information is started from the start-point viewing information, and continuously decrease the transition speed for a given period of time before the transition of viewing information is stopped at the end-point viewing information.

7. The image method according to claim 6, further comprising:

detecting obstacle information including at least a position an obstacle exists; and

receiving the obstacle information, determining whether an obstacle is present on two-dimensional image data to be converted or not, and reducing the interpolation width in the viewing interpolation regulation to decrease the transition speed of viewing information when determining that an obstacle is present on the two-dimensional image data.

8. The image method according to claim 6, further comprising:

detecting a range of the converted two-dimensional image data that is included in the projection surface;

determining whether the converted two-dimensional image data covers the overall directions at the time when the transited viewing information reaches the end-point viewing information, and when determining that the overall directions are not covered, changing the end-point viewing information.