Vehicular image processing apparatus and vehicular image processing system

• US 10,183,621 B2
• Filed: 05/12/2016
• Issued: 01/22/2019
• Est. Priority Date: 11/12/2015
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: Hodohara, Norifumi
• Primary Examiner(s): Rush, Eric
• Application Number:
  US15/153,018
• Publication Number:
  US 20170140542A1
• Time in Patent Office: 985 Days
• Field of Search: 382100, 382103, 382104, 382106, 382154, 382276, 382282, 382284, 382291, 382293, 382294, 348135, 348143, 348148, 348159, 345629, 345634
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/60 characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
• CPC Class Codes: B60R 2300/602 with an adjustable viewpoint
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 2207/30264 Parking
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/61 : the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A vehicular image processing system and a vehicular image processing apparatus are provided which combine an image from a photographing device for detection of an object forward of a vehicle and an image from a photographing device for parking assistance, thereby enabling detection, in a synthesized image, of an obstacle of which the entirety cannot be captured by each photographing device since the object is located in a region where detection is difficult in the conventional art, for example, the object is located at a short distance from the own-vehicle.

Claims

1. A vehicular image processing apparatus comprising:

a receiver to receive input of a first captured image obtained by photographing an area forward of a vehicle and a second captured image obtained by photographing an area around the vehicle, a photographing area of the second captured image partially overlapping that of the first captured image;

a processor comprising;

a projection image converter configured to perform projective transformation on the second captured image at a plurality of projection planes corresponding to different distances from the vehicle and output a projection image group comprising projection images having been captured from the same position as that for the first captured image,

a first image combiner configured to combine each of the projection images and the first captured image, respectively, such that each output image comprising one of the projection images and the first captured image is continuous, and

a detection module configured to detect an obstacle forward of the vehicle from at least one of the output images synthesized by the first image combiner; and

a transmitter to send information of the obstacle to a display.

2. The vehicular image processing apparatus according to claim 1, wherein the first image combiner is further configured to output a synthesized image group corresponding to the plurality of projection planes.

3. The vehicular image processing apparatus according to claim 2, wherein the processor further comprises:

an image subtractor configured to calculate a similarity per pixel for overlapping portions of the first captured image and each projection image of the projection image group; and

a distance identification controller configured to perform comparison per region and perform labeling with a distance of a projection plane, among the plurality of projection planes, for the projection image corresponding to a most similar region, as a distance of the region.

4. The vehicular image processing apparatus according to claim 3, wherein the processor further comprises:

a second image combiner configured to select the projection image of each labeled region and combine the selected projection image with the first captured image, thereby to perform combination such that a boundary between the selected projection image and the first captured image can be seen as continuous.

5. The vehicular image processing apparatus according to claim 2, wherein the processor further comprises:

an image subtractor configured to calculate a difference per pixel for overlapping portions of the first captured image and each projection image of the projection image group;

a distance identification controller configured to integrate and compare difference values per a strip-like region, and perform labeling with a distance of a projection plane, among the plurality of projection planes, corresponding to the strip-like region having a lowest integrated value, as a distance of the strip-like region; and

a strip combiner configured to select the projection image of each labeled strip-like region and combine the selected projection image with the first captured image thereby to perform combination such that a boundary between the selected projection image and the first captured image can be seen continuous.

6. The vehicular image processing apparatus according to claim 1, wherein the first captured image is subjected to the projective transformation.

7. A vehicular image processing system comprising:

a first photographing device mounted on a vehicle and configured to photograph an area forward of the vehicle;

a second photographing device mounted on the vehicle and configured to photograph an area around the vehicle, the area photographed by the second photographing device partially overlapping the area photographed by the first photographing device;

a vehicular image processing apparatus comprising;

a receiver to receive an input of a first captured image obtained by photographing the area forward of the vehicle, from the first photographing device, and a second captured image obtained by photographing the area around the vehicle, from the second photographing device, a photographing area of the second captured image partially overlapping that of the first captured image, and

a processor comprising;

a projection image conversion module configured to perform projective transformation on the second captured image at a plurality of projection planes corresponding to different distances from the vehicle and output a projection image group comprising projection images having been captured from the same position as that for the first captured image,

a first image combination module configured to combine each of the projection images and the first captured image, respectively, such that each output image comprising one of the projection images and the first captured image is continuous, and

a detection determination module configured to detect an obstacle forward of the vehicle from at least one of the output images synthesized by the first image combination module;

and

a display device configured to display a result output from the vehicular image processing apparatus.

8. The vehicular image processing system according to claim 7, wherein the second photographing device is further configured to output an overhead image of the vehicle as viewed from above.