Image generation apparatus

• US 10,703,275 B2
• Filed: 04/20/2017
• Issued: 07/07/2020
• Est. Priority Date: 05/18/2016
• Status: Active Grant

• Current Assignee: Fujitsu Ten Limited (DENSO Corporation)
• Original Assignee: DENSO Ten Limited (f/k/a Fujitsu Ten Ltd.) (DENSO Corporation)
• Inventors: Fujita, Atsushi
• Primary Examiner(s): Chin, Ricky
• Application Number:
  US15/492,313
• Publication Number:
  US 20170334356A1
• Time in Patent Office: 1,174 Days
• Field of Search: None
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/002 specially adapted for covering the peripheral part of the vehicle, e.g. for viewing tyres, bumpers or the like integrated in the windows B60R1/001; combined with rear-view mirrors B60R1/08
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 2300/20 characterised by the type of display used
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/60 characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: B60R 2300/8053 for bad weather conditions or night vision
• CPC Class Codes: B60R 2300/8086 for vehicle path indication
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

An image generation apparatus includes an image processor and a detector. The image processor generates a composite image viewed from at least one virtual viewpoint, and outputs the generated composite image to a display. The detector detects an indication of a driver intention to drive the host vehicle to a right or to a left relative to a travelling direction of the host vehicle. The image processor generates the composite image viewed from a virtual viewpoint located in front of the host vehicle in the travelling direction, the composite image showing a lateral surface of the host vehicle on a side corresponding to the detected indication of the driver intention. Thus, it is possible to make it easier to check a situation at a time of a right turn, a left turn, or a travelling direction change.

Claims

1. An image generation apparatus for use with a user in a host vehicle, the image generation apparatus comprising:

a memory storing a location of at least one virtual viewpoint within a virtual viewpoint region and a location of at least one gaze object within a gaze object region, which are respectively set by an operation of the user in the host vehicle via an input to a navigation or touch screen, the operation including;

(a) setting an angle of a line between the at least one virtual viewpoint and the at least one gaze object, (b) setting a distance between the at least one virtual viewpoint and the at least one gaze object, and (c) setting a height of the at least one virtual viewpoint from a road surface;

an image processor configured to;

generate a composite image viewed from the location of the at least one virtual viewpoint and oriented towards the location of the at least one gaze object of the host vehicle, based on captured images captured by a plurality of image capturing apparatus mounted on the host vehicle, and

output the generated composite image to a display; and

a detector that detects an indication of a driver intention to drive the host vehicle to a right or to a left relative to a travelling direction of the host vehicle, based on a drive operation with the host vehicle, wherein

in a case where the indication of the driver intention has been detected by the detector, the image processor generates the composite image viewed from a virtual viewpoint located in front of a center of the host vehicle in the travelling direction, the composite image showing a lateral surface of the host vehicle on a side corresponding to the detected indication of the driver intention viewed in a direction having a component that is opposite to the travelling direction.

2. The image generation apparatus according to claim 1, wherein

the image processor generates the composite image viewed from an external virtual viewpoint located in front of the host vehicle in the travelling direction, the composite image showing the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

3. The image generation apparatus according to claim 2, wherein

(i) in a case where the indication of the driver intention to drive the host vehicle to the left relative to the travelling direction has been detected, the image processor generates the composite image showing a left lateral surface of the host vehicle viewed from a left external virtual viewpoint located in front of the host vehicle relative to the travelling direction, and (ii) in a case where the indication of the driver intention to drive the host vehicle to the right relative to the travelling direction has been detected, the image processor generates the composite image showing a right lateral surface of the host vehicle viewed from a right external virtual viewpoint located in front of the host vehicle relative to the travelling direction.

4. The image generation apparatus according to claim 1, wherein

the image processor generates the composite image showing the lateral surface of the host vehicle viewed from a location determined beforehand as a location of a side mirror provided to the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

5. The image generation apparatus according to claim 1, wherein

the image processor generates the composite image viewed from a frontal virtual viewpoint located inside the host vehicle in the travelling direction, the composite image showing the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

6. The image generation apparatus according to claim 1, wherein

in a case where the travelling direction of the host vehicle is a backward direction, the image processor generates the composite image viewed from a virtual viewpoint located behind the host vehicle, and outputs the generated composite image to the display along with an image showing an area behind the host vehicle viewed from the host vehicle, the composite image showing the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

7. The image generation apparatus according to claim 1, wherein

the image processor horizontally flips the composite image and outputs the horizontally-flipped composite image to the display.

8. The image generation apparatus according to claim 1, further comprising:

a memory processor that causes;

(i) the virtual viewpoint selected from amongst a plurality of the virtual viewpoints to be associated with the indication of the driver intention and (ii) the associated virtual viewpoint to be stored in a memory, wherein

the image processor generates the composite image viewed from the virtual viewpoint stored in association with the detected indication of the driver intention, the composite image showing the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

9. The image generation apparatus according to claim 1, wherein

the at least one viewpoint includes a location of a side mirror of the host vehicle, and

the at least one gaze object includes a rear side of the host vehicle.

10. An image display system for use with a user in a host vehicle, the image display system comprising:

a display;

a memory storing a location of at least one virtual viewpoint within a virtual viewpoint region and a location of at least one gaze object within a gaze object region, which are respectively set by an operation of the user in the host vehicle via an input to a navigation or touch screen, the operation including;

(a) setting an angle of a line between the at least one virtual viewpoint and the at least one gaze object, (b) setting a distance between the at least one virtual viewpoint and the at least one gaze object, and (c) setting a height of the at least one virtual viewpoint from a road surface; and

an image generation apparatus that includes;

(i) an image processor that generates a composite image viewed from the location of the at least one virtual viewpoint and oriented towards the location of the at least one gaze object, based on captured images captured by a plurality of image capturing apparatus mounted on a host vehicle, and that outputs the generated composite image to the display; and

(ii) a detector that detects an indication of a driver intention to drive the host vehicle to a right or to a left relative to a travelling direction of the host vehicle, based on a drive operation with the host vehicle, wherein

in a case where the indication of the driver intention has been detected by the detector, the image processor generates the composite image viewed from a virtual viewpoint located in front of a center of the host vehicle in the travelling direction, the composite image showing a lateral surface of the host vehicle on a side corresponding to the detected indication of the driver intention viewed in a direction having a component that is opposite to the travelling direction.

11. The image display system according to claim 10, wherein

the at least one viewpoint includes a location of a side mirror of the host vehicle, and

the at least one gaze object includes a rear side of the host vehicle.

12. An image display method for use with a user in a host vehicle, the method comprising the steps of:

(a) setting, by an operation of the user in the host vehicle via an input to a navigation or touch screen, a location of at least one virtual viewpoint within a virtual viewpoint region and a location of at least one gaze object within a gaze object region, the operation including;

setting an angle of a line between the at least one virtual viewpoint and the at least one gaze object,

setting a distance between the at least one virtual viewpoint and the at least one gaze object, and

setting a height of the at least one virtual viewpoint from a road surface;

(b) generating a composite image viewed from the location of the at least one virtual viewpoint and oriented towards the location of the at least one gaze object, based on captured images captured by a plurality of image capturing apparatus mounted on a host vehicle, and outputting the generated composite image to a display; and

(c) detecting an indication of a driver intention to drive the host vehicle to a right or to a left relative to a travelling direction of the host vehicle, based on a drive operation with the host vehicle, wherein

in a case where the indication of the driver intention has been detected by the step (c), the step (b) generates the composite image viewed from a virtual viewpoint located in front of a center of the host vehicle in the travelling direction, the composite image showing a lateral surface of the host vehicle on a side corresponding to the detected indication of the driver intention viewed in a direction having a component that is opposite to the travelling direction.

13. The image display method according to claim 12, wherein

the step (b) generates the composite image viewed from an external virtual viewpoint located in front of the host vehicle in the travelling direction, the composite image showing the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

14. The image display method according to claim 13, wherein

(i) in a case where the indication of the driver intention to drive the host vehicle to the left relative to the travelling direction of the host vehicle has been detected, the step (b) generates the composite image showing a left lateral surface of the host vehicle viewed from a left external virtual viewpoint located in front of the host vehicle relative to the travelling direction, and (ii) in a case where the indication of the driver intention to drive the host vehicle to the right relative to the travelling direction has been detected, the step (b) generates the composite image showing a right lateral surface of the host vehicle viewed from a right external virtual viewpoint located in front of the host vehicle relative to the travelling direction.

15. The image display method according to claim 12, wherein

the step (b) generates the composite image showing the lateral surface of the host vehicle viewed from a location determined beforehand as a location of a side mirror provided to the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

16. The image display method according to claim 12, wherein

the step (b) generates the composite image viewed from a frontal virtual viewpoint located inside the host vehicle in the travelling direction, the composite image showing the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

17. The image display method according to claim 12, wherein

in a case where the travelling direction of the host vehicle is a backward direction, the step (b) generates the composite image viewed from a virtual viewpoint located behind the host vehicle, and outputs the generated composite image to the display along with an image showing an area behind the host vehicle viewed from the host vehicle, the composite image showing the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

18. The image display method according to claim 12, wherein

the step (b) horizontally flips the composite image and outputs the horizontally-flipped composite image to the display.

19. The image display method according to claim 12, further comprising the step of:

(c) causing;

(i) the virtual viewpoint selected from amongst a plurality of the virtual viewpoints to be associated with the indication of the driver intention and (ii) the associated virtual viewpoint to be stored in a memory, wherein

the step (b) generates the composite image viewed from the virtual viewpoint stored in association with the detected indication of the driver intention, the composite image showing the lateral surface of the host vehicle on the side corresponding to the detected indication of the driver intention.

20. The image display method according to claim 12, wherein

the at least one viewpoint includes a location of a side mirror of the host vehicle, and

the at least one gaze object includes a rear side of the host vehicle.