Image capturing device and method, program, and record medium to perform exposure control based on the brightness in an attention area corresponding to a detected object

• US 10,453,214 B2
• Filed: 11/12/2015
• Issued: 10/22/2019
• Est. Priority Date: 01/22/2015
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: Fujita, Takeo, Yamashita, Koichi, Matoba, Narihiro
• Primary Examiner(s): Cutler, Albert H
• Application Number:
  US15/539,233
• Publication Number:
  US 20180012374A1
• Time in Patent Office: 1,440 Days
• Field of Search: 348362, 348364
• CPC Class Codes: G03B 15/03 Combinations of cameras with lighting apparatus; Flash units
• CPC Class Codes: G03B 7/091 Digital circuits
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/73 by influencing the exposure time

Abstract

An object having a high attention degree is selected from objects detected by a detection means, brightness of a captured image is calculated by using an attention region corresponding to the selected object as a detection frame, and exposure control is performed based on the calculated brightness. The attention degree is evaluated higher with the decrease in the distance. Alternatively, the attention degree is evaluated higher as the direction becomes closer to the traveling direction. The attention region is made larger with the decrease in the distance to the object. It is also possible to judge the type of the object and determine the size of the attention region based on the result of the judgment. A subject to be paid attention to is made clearly visible.

Claims

1. An image capturing device comprising:

a camera configured to capture an image of a subject situated in an imaging field angle range and thereby generates a captured image;

a radar configured to detect objects existing in a detection range at least partially overlapping the imaging field angle range and produce information representing a position of each of the detected objects, information representing a direction of each of the detected objects and information representing a distance to each of the detected objects;

a computer processor; and

a memory storing instructions which, when executed, causes the computer processor to perform a process of

evaluating an attention degree of each of the detected objects based on at least one of the information representing the position of each of the detected objects, the information representing the distance to each of the detected objects, and the information representing the direction of each of the detected objects, which are produced by the radar, and selecting one or more objects based on the attention degree;

specifying one or more regions in the captured image respectively including the selected objects as attention regions based on the attention degree, said specified attention regions excluding each detected object not selected;

calculating a brightness of the captured image in the specified attention regions; and

performing exposure control on the camera based on the calculated brightness,

wherein;

the process selects a plurality of objects based on the information representing the direction of the object and the information representing the distance,

the process specifies the attention region for each of the plurality of objects selected,

the camera captures video of the subject and outputs the captured image every frame period,

the process calculates the brightness in each frame period by using one of the plurality of attention regions respectively specified for the plurality of objects as a detection frame, and

the process adjusts an exposure condition in a certain frame period based on the brightness calculated by using one of the plurality of attention regions as the detection frame and uses the adjusted exposure condition later in the image capturing in a frame period in which the brightness is calculated by using the same attention region as the detection frame.

2. The image capturing device according to claim 1, wherein the process evaluates an attention degree of the object higher with the decrease in the distance to the object represented by the information representing the distance regarding each detected object outputted from the radar and makes the object selection based on the attention degree.

3. The image capturing device according to claim 1, wherein:

the camera is mounted on a vehicle, and

the process further comprises detecting a traveling direction of the vehicle from a steering direction of a steering wheel, and producing information representing the traveling direction, and

the process makes the object selection based on the information representing the traveling direction and the information representing the direction of the object and the information representing the distance to the object.

4. The image capturing device according to claim 3, wherein the process evaluates an attention degree of the object higher as the direction of the object represented by the information representing the direction regarding the object becomes closer to the traveling direction, and makes the object selection based on the attention degree.

5. The image capturing device according to claim 1, wherein the process specifies a region in the captured image including a part estimated to be occupied by the selected object in a case where the object is assumed to be of the largest type among a presumed plurality of types of objects as the attention region.

6. The image capturing device according to claim 1, wherein the process calculates the brightness by using the plurality of attention regions respectively specified for the plurality of objects as a single detection frame.

7. The image capturing device according to claim 1, wherein the process further comprises judging a type of the selected object based on the captured image,

wherein the process determines the attention region based on the judged type of the selected object.

8. The image capturing device according to claim 7, wherein the process judges the type of the selected object by analyzing a part in the captured image that is estimated to be occupied by the selected object in a case where the selected object is assumed to be of the largest type among a presumed plurality of types of objects.

9. An image capturing device comprising:

a camera configured to capture an image of a subject situated in an imaging field angle range and thereby generates a captured image;

a radar configured to detect an object existing in a detection range at least partially overlapping the imaging field angle range and produce information representing a direction of the object and information representing a distance to the object;

a computer processor; and

a memory storing instructions which, when executed, causes the computer processor to perform a process of

selecting one or more objects based on the information representing the direction of the object and the information representing the distance to the object which are produced by the radar, and producing information representing a position of the selected object in the captured image and information representing the distance to the selected object;

specifying a region in the captured image including the selected object as an attention region based on the information representing the position of the selected object and the information representing the distance to the selected object;

calculating brightness of the captured image in the specified attention region; and

performing exposure control on the camera based on the calculated brightness, wherein;

the camera is mounted on a vehicle,

the process further comprises detecting a position of a lane marking and a median in the captured image based on the captured image, and producing information representing the detected position of the lane marking and the median, and

the process makes the object selection based on the information representing the position of the lane marking and the median and the information representing the direction of the object and the information representing the distance.

10. An image capturing method in an image capturing device, the image capturing device including:

a camera configured to capture an image of a subject situated in an imaging field angle range and thereby generates a captured image; and

a radar configured to detect objects existing in a detection range at least partially overlapping the imaging field angle range, and produce information representing a position of each of the detected objects, information representing a direction of each of the detected objects and information representing a distance to each of the detected objects, the method comprising;

evaluating an attention degree of each of the detected objects based on at least one of the information representing a position of each of the detected objects, the information representing the distance to each of the detected objects, and the information representing the direction of each of the detected objects, which are produced by the radar, and selecting one or more objects based on the attention degree;

specifying one or more regions in the captured image respectively including the selected objects as attention regions based on the attention degree, said specified attention regions excluding each detected object not selected;

calculating a brightness of the captured image in the specified attention regions; and

performing exposure control on the camera based on the calculated brightness,

wherein;

the method selects a plurality of objects based on the information representing the direction of the object and the information representing the distance,

the method specifies the attention region for each of the plurality of objects selected,

the camera captures video of the subject and outputs the captured image every frame period,

the method calculates the brightness in each frame period by using one of the plurality of attention regions respectively specified for the plurality of objects as a detection frame, and

the method adjusts an exposure condition in a certain frame period based on the brightness calculated by using one of the plurality of attention regions as the detection frame and uses the adjusted exposure condition later in the image capturing in a frame period in which the brightness is calculated by using the same attention region as the detection frame.

11. A non-transitory computer-readable record medium storing a program for causing a computer to execute the image capturing method according to claim 10.