Multi-lens camera system, vehicle mounting the multi-lens camera system, and range-finding method executed by the multi-lens camera system

• US 9,420,151 B2
• Filed: 03/18/2013
• Issued: 08/16/2016
• Est. Priority Date: 03/21/2012
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Yokota, Soichiro
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): MESSMORE, JONATHAN R
• Application Number:
  US13/845,860
• Publication Number:
  US 20130250109A1
• Time in Patent Office: 1,247 Days
• Field of Search: 382/224, 348/148
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

A multi-lens camera system includes multiple camera units including respective optical filters and image sensors that acquire a captured image via the optical filter, each of the optical filters having multiple filter regions whose optical characteristics differ respectively, and an image processor, operably connected to the multiple camera unit, to execute different types of image processing on the captured image to form an image that contains multiple image regions whose characteristics differ depending on the optical characteristics of the filter region in which they are acquired.

Claims

1. A multi-lens camera system comprising:

multiple camera units, the multiple camera units include multiple optical filters and multiple image sensors, the multiple image sensors capture images via the multiple optical filters, each of the multiple optical filters include multiple filter regions, each of the multiple filter regions include different optical characteristics respectively, the multiple filter regions include a clear filter region which corresponds to a central image region of the captured images and a color filter region which corresponds to an upper image region of the captured images; and

an image processor operably connected to the multiple camera units, the image processor configured to,

execute different types of image processing on the captured images to form an image that contains multiple image regions, each of the multiple image regions include different characteristics depending on the optical characteristics of the multiple filter regions, the multiple image regions include the central image region and the upper image region,

perform a parallax calculation for the clear filter region of the central image region of the captured images, and

recognize a target in the color filter region of the upper image region of the captured images based on color information.

2. The multi-lens camera system of claim 1, wherein the image processor is configured to recognize the target in the captured images by multiple different image recognitions, the multiple different image recognitions differing from each other depending on the multiple image regions.

3. The multi-lens camera system of claim 2, wherein the image processor is configured to recognize the target in the captured images using the parallax information obtained between the central image region in the captured images and color information in the upper image region.

4. The multi-lens camera system of claim 1, wherein the image processor is configured to recognize another target in a lower image region of the captured images using the parallax information obtained between the central image region in the captured images and a luminance information in the lower image region.

18. The multi-lens camera system of claim 1, wherein the target recognized in the color filter region of the upper image region by the image processor includes at least one of traffic signal and traffic sign.

19. The multi-lens camera system of claim 1, wherein the parallax calculation is configured to be performed by a hardware, and recognition of a target is configured to be performed by executing a set of computer readable instructions.

20. The multi-lens camera system of claim 1, wherein the image processor is configured to perform the parallax calculation also for the color filter region, and the parallax calculation for the color filter region is performed only on the target recognized in the color filter region of the upper image region of the captured image.

5. A vehicle comprising:

a multi-lens camera system including,

multiple camera units, the multiple camera units include multiple optical filters and multiple image sensors, the multiple image sensors acquire captured images via the multiple optical filters, each of the multiple optical filters include multiple filter regions, each of the multiple filter regions include optical characteristics that differ from each other respectively, the multiple filter regions include a clear filter region which corresponds to a central image region of the captured images, and

an image processor, the image processor operably connected to the multiple camera units, the image processor configured to,

process the captured images into multiple image regions, the multiple image regions includes the central image region, each of the multiple image regions include characteristics that differ depending on the optical characteristics of the multiple filter regions, the differing characteristics of the multiple image regions configured to recognize various types of targets that are positioned ahead of a vehicle, and

perform a parallax calculation for the clear filter region of the central image region of the captured images; and

a vehicle drive control unit, the vehicle drive control unit configured to control steering and braking of the vehicle based on a recognition result provided by the image processor.

6. The vehicle of claim 5, wherein the image processor is further configured to divide the captured images into the multiple image regions, the multiple image regions includes the central image region where an object in the central image region is recognized as a target using the parallax calculation.

7. The vehicle of claim 6, wherein the image processor is further configured to,

recognize a dividing line on a surface and the target in the central region in the captured images, wherein recognizing the dividing line based on a luminance information of a lower image region, and recognizing the target in the central image region of the captured images in three dimensions;

calculate the parallax information for the central image region in the captured images and a height of the surface, wherein calculating the parallax information based on the luminance information of the lower image region, and calculating the height of the surface based on the dividing line;

perform clustering results using the parallax information; and

modify the clustering results based on a size of the object in the central image region that is recognized as the target using the parallax calculation, the object in the central image region includes at least one of another vehicle ahead of the vehicle, a pedestrian, a motorbike and a bicycle.

8. The vehicle of claim 7, wherein the image processor is configured to divides the captured images into the multiple image regions, the multiple image regions includes an upper image region where an object in the upper image region is recognized as the target using color information of the upper image region, the object in the upper image region includes at least one of a traffic light and a traffic sign.

9. The vehicle of claim 8, wherein the image processor is configured to,

recognize the object in the upper image region using color information of the upper image region in three dimensions during calculation of the parallax information and clustering results using the parallax information for the object in the central region in the captured images,

calculate parallax information only for the object in the upper image region using the color information of the upper image region during modification of the clustering results based on the size of the object in the central region that is recognized as the target using the parallax calculation.

10. The vehicle of claim 7, wherein the image processor is configured to divide the captured images into the multiple image regions, the multiple image regions includes the lower image region where an object in the lower image region is detected using the luminance information, the object in the lower image region is a raindrop.

11. The vehicle of claim 10, wherein the image processor is configured to detect the object in the lower image region during calculation of the parallax information for the central image region in the captured images and modifying of the clustering results using the parallax information for the central image region in the captured images.

12. A range-finding method in a multi-lens camera system that includes multiple camera units, the multiple camera units include multiple optical filters and multiple image sensors, the multiple image sensors capture images via the multiple optical filters, each of the multiple optical filters include multiple filter regions whose optical characteristics differ respectively, the method comprising:

dividing each of the captured images into multiple image regions, each of the multiple image regions include characteristics that differ depending on the optical characteristics of the multiple filter regions, the multiple image regions includes a first image region and a second image region;

recognizing a first target object and a second target object, the first target object is recognized in the first image region of the captured image in three dimensions, the second target object is recognized in the second image region of the captured image in three dimensions;

calculating parallax information for the first image region in the captured image and parallax information for the second image region in the captured image, the parallax information for the first image region is calculated while subsequently recognizing the second target object in the second image region, and parallax information for the second image region is only calculated when the second target object is recognized in the second image region of the captured image;

performing clustering results using the parallax information for the first image region, wherein recognizing of the second target object in the second image region is performed during the clustering results for the first image region; and

modifying the clustering results for the first image region, wherein calculating parallax information only for the second target object in the second image region is performed during modifying the clustering results for the first image region.

13. The range-finding method of claim 12, further comprising:

detecting an object in a third region of the captured images, the object in the third region is detected during calculating the parallax information for the first image region and performing clustering results using the parallax information for the first image region.

14. The method of claim 12, wherein the multiple image regions include an upper image region, the upper image region corresponds to a color filter region.

15. The method of claim 12, wherein the multiple image regions include a lower image region, the lower image region corresponds to a neutral density filter region.

16. The method of claim 12, wherein modifying the clustering results based on a size of the first target object recognized in the first image region of the captured images.

17. The method of claim 16, wherein the first target object includes at least one of a vehicle in front, a pedestrian, a motorbike, and a bicycle.