IMAGE GENERATING APPARATUS, IMAGE GENERATING METHOD, AND RECORDING MEDIUM

• US 20170352275A1
• Filed: 05/30/2017
• Published: 12/07/2017
• Est. Priority Date: 06/07/2016
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: MARUYAMA, AYAKO, NOBORI, KUNIO, AZUMA, TAKEO, SATO, SATOSHI, WAKAI, NOBUHIKO
• Granted Patent: US 11,056,001 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 1/003 for viewing trailer hitches
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An image generating apparatus generates an image to be displayed on a display and includes at least one memory and a control circuit. The control circuit acquires a plurality of camera images captured by a plurality of cameras installed in a vehicle, calculates a distance between one of the cameras and a target to be projected in the camera images, detects a position of a light-transmissive object or a reflective object in the camera images, and generates an image from a point of view that is different from points of view of the plurality of camera images by using the plurality of camera images and the distance, the generated image including a predetermined image that is displayed at the position of the light-transmissive object or the reflective object.

Claims

1. An image generating apparatus that generates an image to be displayed on a display, the image generating apparatus comprising:

at least one memory; and

a control circuit,

wherein the control circuit

(a) acquires a plurality of camera images captured by a plurality of cameras installed in a vehicle,

(b) calculates a distance between one of the cameras and a target included in the camera images in a three-dimensional space by using the plurality of camera images,

(c) detects a position of a light-transmissive object or a reflective object in the camera images, and

(d) generates an image from a point of view that is different from points of view of the plurality of camera images by using at least one of the plurality of camera images and the distance, the generated image including a predetermined image that is displayed at the position of the light-transmissive object or the reflective object.

2. The image generating apparatus according to claim 1,

wherein the control circuit further

(e) estimates a distance between the camera and the light-transmissive object or the reflective object, and

(f) corrects the distance between the camera and the target at the position of the light-transmissive object or the reflective object in the camera images to the estimated distance, and

wherein in the (d), the image is generated by using the plurality of camera images and the corrected distance.

3. The image generating apparatus according to claim 1,

wherein in the (a), a polarized camera image captured by a polarization camera that captures an image through a polarization plate is acquired, and

wherein in the (c), a position where brightness of the polarized camera image periodically changes is detected as the position of the light-transmissive object or the reflective object.

4. The image generating apparatus according to claim 1,

wherein the control circuit further

(g) acquires a plurality of infrared camera images captured by a plurality of far-infrared cameras, and

(h) calculates a distance between one of the far-infrared cameras and the target included in the infrared camera images in a three-dimensional space by using the plurality of infrared camera images, and

wherein in the (c), a position where a difference between the distance calculated by using the camera images and the distance calculated by using the infrared camera images is larger than a predetermined threshold is detected as the position of the light-transmissive object or the reflective object.

5. The image generating apparatus according to claim 1, wherein in the (c), a frame-shaped object included in the camera images is recognized, and a region inside the recognized frame-shaped object is detected as the position of the light-transmissive object or the reflective object.

6. The image generating apparatus according to claim 1, wherein in the (d), a part of one of the plurality of camera images is displayed at the position of the light-transmissive object or the reflective object in the generated image.

7. The image generating apparatus according to claim 1, wherein in the (d), a camera image including a larger area of the light-transmissive object or the reflective object is preferentially displayed among the plurality of camera images at the position of the light-transmissive object or the reflective object in the generated image.

8. The image generating apparatus according to claim 7, wherein in the (d), a camera image including a larger area of the light-transmissive object or the reflective object is preferentially displayed among the plurality of camera images at positions of a plurality of light-transmissive objects or reflective objects in the generated image.

9. The image generating apparatus according to claim 1,

wherein the control circuit further

(i) separates a light-transmissive component and a reflective component from each other, the light-transmissive component and the reflective component being included at the position of the light-transmissive object in the camera images, and

wherein in the (d), the light-transmissive component and the reflective component are displayed by being assigned weights at a predetermined ratio at the position of the light-transmissive object in the generated image.

10. The image generating apparatus according to claim 9, wherein in the (i), a polarized camera image captured by a polarization camera that captures an image through a polarization plate is acquired, and the light-transmissive component and the reflective component are separated from each other by using the polarized camera image.

11. The image generating apparatus according to claim 9, wherein in the (d), the predetermined ratio is received from a user, and the light-transmissive component and the reflective component are displayed by being assigned weights at the predetermined ratio received from the user at the position of the light-transmissive object in the generated image.

12. The image generating apparatus according to claim 9, wherein in the (d), the light-transmissive component and the reflective component are displayed at the position of the light-transmissive object in the generated image by being assigned weights in such a manner that the weight of the reflective component is lower than the weight of the light-transmissive component when external light is brighter.

13. The image generating apparatus according to claim 1, wherein in the (d), if a predetermined condition that the camera images exhibit an appropriate exposure state is not satisfied, such a message is displayed in the generated image that prompts a driver to see a periphery of the vehicle.

14. The image generating apparatus according to claim 1,

wherein the plurality of cameras include a first camera that captures an image of a forward area of the vehicle and a second camera that captures an image of a backward area of the vehicle,

wherein the control circuit further

(j) acquires a heading direction of the vehicle, and

wherein in the (c), the position of the light-transmissive object or the reflective object is detected by using a camera image obtained by a camera that captures an image in a direction that is same as the heading direction of the vehicle, the camera being selected from the first camera and the second camera.

15. The image generating apparatus according to claim 2, wherein in the (c), if the estimated distance between the camera and the light-transmissive object or the reflective object is larger than a predetermined threshold, the distance between the camera and the target at the position of the light-transmissive object or the reflective object is not corrected.

16. The image generating apparatus according to claim 2, wherein in the (c), if a detected lowest end of the light-transmissive object or the reflective object is higher than or equal to a predetermined threshold from a surface of a road, the distance between the camera and the light-transmissive object or the reflective object is not estimated, and the distance between the camera and a point in a space to be projected at the position of the light-transmissive object or the reflective object is not corrected.

17. The image generating apparatus according to claim 2, wherein in the (c), if a detected size of the light-transmissive object or the reflective object is smaller than a predetermined threshold, the distance between the camera and the light-transmissive object or the reflective object is not estimated, and the distance between the camera and a point in a space to be projected at the position of the light-transmissive object or the reflective object is not corrected.

18. An image generating method for generating an image to be displayed on a display installed in a vehicle, the image generating method comprising:

(a) acquiring a plurality of camera images captured by a plurality of cameras installed in the vehicle,

(b) calculating a distance between one of the cameras and a point in a space to be projected in the camera images by using the plurality of camera images,

(c) detecting a position of a light-transmissive object or a reflective object in the camera images, and

(d) generating an image from a point of view that is different from the points of view of the plurality of camera images by using the plurality of camera images and the distance, the generated image including a predetermined image that is displayed at the position of the light-transmissive object or the reflective object.

19. A recording medium storing a control program for causing an apparatus including a processor to execute a process, the recording medium being non-volatile and computer readable,

wherein the process is a process for generating an image to be displayed on a display installed in the vehicle, the process including

(a) acquiring a plurality of camera images captured by a plurality of cameras installed in a vehicle,

(b) calculating a distance between one of the cameras and a point in a space to be projected in the camera images by using the plurality of camera images,

(c) detecting a position of a light-transmissive object or a reflective object in the camera images, and

(d) generating an image from a point of view that is different from the points of view of the plurality of camera images by using the plurality of camera images and the distance, the generated image including a predetermined image that is displayed at the position of the light-transmissive object or the reflective object.

20. An image capturing apparatus comprising:

a first visible light camera that captures a first image including an image of an object;

a second visible light camera that captures a second image including an image of the object, each of the first visible light camera and the second visible light camera having sensitivity to visible light more than infrared rays;

a first infrared camera that captures a third image including an image of the object;

a second infrared camera that captures a fourth image including an image of the object, each of the first infrared camera and the second infrared camera having sensitivity to infrared rays more than visible light;

a non-transitory memory that stores in advance camera parameters including external parameters regarding physical arrangement of the first visible light camera, the second visible light camera, the first infrared camera, and the second infrared camera, and internal parameters regarding characteristics of the first visible light camera, the second visible light camera, the first infrared camera, and the second infrared camera; and

a processor,

wherein the processor calculates (i) a first three dimensional position on a point of the object, using a first point in the first image, a second point in the second image, and the camera parameters, and (ii) a second three dimensional position on the point of the object, using a third point in the third image, a fourth point in the fourth image, and the camera parameters,

wherein the processor calculates (iii) a first length between a fifth point and the first three dimensional position and (iv) a second length between the fifth point and the second three dimensional position,

wherein when a difference between the first length and the second length is bigger than a predetermined value, the processor determines a glass is provided between the object and the first visible light camera, the second visible light camera, the first infrared camera, and the second infrared camera.