Image processing device

• US 8,330,816 B2
• Filed: 11/19/2007
• Issued: 12/11/2012
• Est. Priority Date: 02/21/2007
• Status: Expired due to Fees

• Current Assignee: Alpine Electronics Incorporated (Alps Alpine Co., Ltd.)
• Original Assignee: Alpine Electronics Incorporated (Alps Alpine Co., Ltd.)
• Inventors: Imamura, Kosuke
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): KATSIKIS, KOSTAS J
• Application Number:
  US11/942,386
• Publication Number:
  US 20080198226A1
• Time in Patent Office: 1,849 Days
• Field of Search: 348/148, 348/E07.085, 382/103, 396/153
• US Class Current: 348/148
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

The present invention provides an image processing device capable of enabling accurate recognition of a solid object present near a vehicle and displaying the solid object. The image processing device includes a viewpoint conversion unit for receiving data of images captured by at least one image capturing camera and generating a top view image, a solid object extraction unit for detecting a solid object from the data of the images captured by the at least one image capturing camera and extracting the solid object, a solid object image generation unit for generating a solid object image in accordance with the solid object extracted by the solid object extraction unit, and an image synthesis unit for synthesizing the solid object image generated by the solid object image generation unit with the top view image generated by the viewpoint conversion unit.

Claims

1. An image processing device configured to display on a display screen, an image of corresponding surroundings of a vehicle, wherein the image processing device comprises:

at least one image capturing camera, wherein each corresponding one image capturing cameras captures images of the corresponding surroundings of the vehicle and generates a corresponding plurality of frames of image data belonging to the corresponding one image capturing camera;

a control unit;

a viewpoint converter, in the control unit, configured to perform viewpoint conversion on the image data to generate a viewpoint-converted image belonging to the corresponding one image capturing camera having a point of view looking downwardly from above the vehicle;

a solid object extractor, in the control unit, configured to extract an outline of a solid object belonging to the corresponding one image capturing camera based on inter-frame differences from the corresponding frames of image data when there is relative motion between the vehicle and the solid object, wherein the solid object extractor further extracts at least one of a moving direction, a size, and a distance from the vehicle to the solid object;

a solid object image generator, in the control unit, configured to generate a solid object image belonging to the corresponding one image capturing camera based on the extracted outline of the solid object, wherein the solid object image generator generates the solid object image in accordance with at least one of the extracted moving direction, the size, and the distance from the vehicle to the solid object;

an image synthesizer, in the control unit, configured to synthesize a display image belonging to the corresponding one image capturing camera based on the generated solid object image and the viewpoint-converted image, wherein the image synthesizer determines the direction and the size of the solid object image in accordance with at least one of the moving direction, the size, and the distance from the vehicle to the solid object extracted by the solid object extractor; and

a display controller, coupled to the control unit, configured to display on the display screen at least the display image belonging to the corresponding one image capturing camera synthesized by the image synthesizer.

2. The image processing device according to claim 1, wherein the solid object image generator generates one of a solid object image directly showing the extracted solid object, a solid object image emphasizing the outline of the solid object, and a solid object image with a shadow.

3. The image processing device according to claim 1, wherein the image synthesizer determines a synthesis position of the solid object image on the basis of a particular solid object included in the viewpoint-converted image.

4. The image processing device according to claim 1,

wherein the at least one image capturing camera includes a plurality of image capturing cameras which captures images of a front side, a rear side, and lateral sides of the vehicle.

5. The image processing device according to claim 1, wherein the image synthesizer determines the color of the solid object image to distinguish the color from the color of a surrounding image.

6. The image processing device according to claim 5,

wherein a color having a large contrast difference from the color of the surrounding image is determined as the color of the solid object image.

7. A navigation system, which comprises:

a navigation device;

a control unit; and

an image processing device which displays on a display screen an image of surroundings of a vehicle, the image processing device comprises;

at least one image capturing camera wherein each corresponding one image capturing cameras captures images of the corresponding surroundings of the vehicle and generates a plurality of frames of image data corresponding to the captured images belonging to the corresponding one image capturing camera,

a viewpoint converter, in the control unit, configured to perform viewpoint conversion on the image data to generate a viewpoint-converted image belonging to the corresponding one image capturing camera having a point of view looking downwardly from above the vehicle,

a solid object extractor, in the control unit, configured to extract an outline of a solid object belonging to the corresponding one image capturing camera based on inter-frame differences from the corresponding frames of image data when there is relative motion between the vehicle and the solid object, wherein the solid object extractor further extracts at least one of a moving direction, a size, and a distance from the vehicle to the solid object;

a solid object image generator, in the control unit, configured to generate a solid object image belonging to the corresponding one image capturing camera based on the extracted outline of the solid object, wherein the solid object image generator generates the solid object image in accordance with at least one of the extracted moving direction, the size, and the distance from the vehicle to the solid object,

an image synthesizer, in the control unit, configured to synthesize a display image belonging to the corresponding one image capturing camera based on the generated solid object image and the viewpoint-converted image, wherein the image synthesizer determines the direction and the size of the solid object image in accordance with at least one of the moving direction, the size, and the distance from the vehicle to the solid object extracted by the solid object extractor and

a display controller configured to display on the display screen at least the display image belonging to the corresponding one image capturing camera synthesized by the image synthesizer.

8. The navigation system according to claim 7,

wherein the navigation device includes a control unit that determines at least one of:

whether or not the vehicle is stopped and whether or not the vehicle is running at a predetermined low speed, and

wherein the display controller displays the synthesized image when the control unit determines that the vehicle is stopped or running at the low speed.

9. A non-transitory computer readable medium storing an image processing program which causes at least an image processing device performing image processing functions comprising displaying on a display an image of surroundings of a vehicle, wherein the image processing functions comprising:

capturing images of the surroundings of the vehicle belonging to a corresponding one image capturing camera and extracting a solid object which has a relative speed with respect to the vehicle included in data of the captured images or with respect to a surrounding environment, and further including extracting at least one of a moving direction, a size, and a distance from the vehicle to the solid object;

generating a solid object image belonging to the corresponding one image capturing camera in accordance with the extracted solid object, and further including generating the solid object image in accordance with the extracted moving direction, size, and distance from the vehicle to the solid object;

performing viewpoint conversion on the data of the captured images belonging to the corresponding one image capturing camera to convert the data into an image looking down from a point of view above the vehicle;

synthesizing the solid object image with the viewpoint-converted image, and further including determining a direction of the solid object image in accordance with the extracted moving direction of the solid object; and

displaying at least the synthesized image belonging to the corresponding one image capturing camera on the display.

10. The non-transitory computer readable medium according to claim 9,

wherein the extracting includes extracting the outline of the solid object, and

wherein the generating includes generating the solid object image in accordance with the extracted outline of the solid object.

11. The non-transitory computer readable medium according to claim 9,

wherein the generating includes generating one of a solid object image directly showing the extracted solid object, a solid object image emphasizing the outline of the solid object, and a solid object image with a shadow.

12. The non-transitory computer readable medium according to claim 9,

wherein the synthesizing includes determining the color of the solid object image to distinguish the color from the color of a surrounding image.

13. The non-transitory computer readable medium according to claim 9,

wherein the synthesizing includes determining a synthesis position of the solid object image on the basis of a particular solid object included in the viewpoint-converted image.

14. The non-transitory computer readable medium according to claim 9,

wherein the viewpoint-converting includes performing the viewpoint conversion on the data of the captured images of a front side, a rear side, and lateral sides of the vehicle to generate omnidirectional data of the captured images centering on the vehicle.

15. An image display method configured to display on a display screen an image of surroundings of a vehicle, the image display method comprising:

capturing a plurality of images of the surroundings of the vehicle belonging to a corresponding one image capturing camera and extracting an outline of a solid object data of the captured images based on inter-frame differences;

generating a solid object image belonging to the corresponding one image capturing camera based on the extracted outline of the solid object based on inter-frame differences when there is relative motion between the vehicle and the solid object;

extracting at least one of a moving direction, a size, and a distance from the vehicle to the solid object and determining a moving direction of the solid object;

performing viewpoint conversion on the data of the at least one captured image to generate a viewpoint-converted image belonging to the corresponding one image capturing camera having a point of view looking downwardly from above the vehicle;

synthesizing a display image belonging to the corresponding one image capturing camera on the display based on the solid object image and the viewpoint-converted image; and

displaying at least the synthesized display image on the display.