System for occlusion adjustment for in-vehicle augmented reality systems

• US 10,248,196 B2
• Filed: 08/23/2016
• Issued: 04/02/2019
• Est. Priority Date: 08/23/2016
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Sisbot, Emrah Akin, Oguchi, Kentaro
• Primary Examiner(s): Wang, Jin Cheng
• Application Number:
  US15/244,975
• Publication Number:
  US 20180059779A1
• Time in Patent Office: 952 Days
• Field of Search: 382154, 345629, 345633
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/205 using a head-up display
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/308 by overlaying the real scene, e.g. through a head-up display on the windscreen
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

The disclosure includes embodiments for providing occlusion adjustment for an in-vehicle augmented reality system. A system may include a three-dimensional heads-up display unit (“3D HUD”) installed in a vehicle. The system may include a memory storing instructions that, when executed, cause the system to: display, on the 3D HUD, a first graphic that is viewable by a driver of the vehicle as overlaying the interested object when the driver is looking at the 3D HUD; determine that at least a portion of the interested object is occluded by an occluding object; turn off the first graphic so that the first graphic is not displayed on the 3D HUD; and display, on the 3D HUD, a second graphic that does not overlay the occluding object and visually indicates to the driver the location of the interested object behind the occluding object when the driver is looking at the 3D HUD.

Claims

1. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer of a vehicle causes the computer to:

determine a type of interested object in an environment external to the vehicle using object priors describing known objects having known types;

determine an importance value associated with the type of interested object;

display an initial graphic on an electronic display of the vehicle;

determine that at least a portion of the interested object is occluded by an occluding object;

determine, based on the importance value, that the importance value meets or exceeds an importance threshold and

responsive to a determination that the importance value meets or exceeds the importance threshold (1) turn off the initial graphic so that the initial graphic is not displayed on the electronic display and (2) display, on the electronic display, a non-occluding graphic that does not overlay the occluding object or the interested object and that visually indicates to a driver a location of the interested object behind the occluding object when the driver is looking at the electronic display.

2. A computer-implemented method comprising:

determining a type of interested object in an environment external to a vehicle using object priors describing known objects having known types;

determining an importance value associated with the type of interested object;

displaying, on an electronic display installed in the vehicle, an initial graphic that is viewable by a driver of the vehicle when looking at the electronic display;

determining that at least a portion of an interested object is occluded by an occluding object;

determining that the importance value meets or exceeds an importance threshold;

responsive to a determination that the importance value meets or exceeds the importance threshold, turning off the initial graphic so that the initial graphic is not displayed on the electronic display; and

displaying, on the electronic display, a non-occluding graphic that does not overlay the occluding object or the interested object and that visually indicates to the driver that the interested object is located behind the occluding object when the driver is looking at the electronic display.

3. The method of claim 2, wherein determining, based on the importance value, that the interested object is not allowed to be occluded by the occluding object includes determining that the importance value fails to meet or exceed an importance threshold.

4. The method of claim 2, wherein the non-occluding graphic is an arrow located above the interested object that indicates to the driver a location of the interested object behind the occluding object.

5. The method of claim 2, wherein the type of interested object is also determined based on sensor data.

6. The method of claim 2, further comprising tracking motion of the interested object using one or more vehicle sensors.

7. The method of claim 6, wherein the one or more vehicle sensors includes a camera that tracks the motion of the interested object.

8. The method of claim 7, wherein the camera tracks the motion of the interested object relative to the occluding object and the vehicle.

9. The method of claim 6, wherein the one or more vehicle sensors includes a range finder that tracks the motion of the interested object and one or more distances of the interested object relative to the occluding object and the vehicle.

10. The method of claim 6, wherein the vehicle sensors detect the interested object moving from a first position to a second position and the method further comprises repositioning a location of the non-occluding graphic as displayed on the electronic display to correspond to the second position wherein the non-occluding graphic was previously displayed on the electronic display to correspond to the first position.

11. A system comprising:

an electronic display installed in a vehicle; and

a memory storing instructions that, when executed, cause the system to;

determine a type of interested object in an environment external to the vehicle using object priors describing known objects having known types;

determine an importance value associated with the type of interested object;

display, on the electronic display, an initial graphic that is viewable by a driver of the vehicle as overlaying an interested object when the driver is looking at the electronic display;

determine that at least a portion of the interested object is occluded by an occluding object;

determine that the importance value meets or exceeds an importance threshold;

responsive to a determination that the importance value meets or exceeds the importance threshold, turn off the initial graphic so that the initial graphic is not displayed on the electronic display;

display, on the electronic display, a non-occluding graphic that does not overlay the occluding object or the interested object and that visually indicates to the driver that the interested object is located behind the occluding object when the driver is looking at the electronic display;

determine that the interested object is no longer occluded by the occluding object;

responsive to determining that the interested object is not occluded, turn off the non-occluding graphic so that the non-occluding graphic is not displayed on the electronic display; and

display, on the electronic display, the initial graphic, wherein the initial graphic is viewable by the driver of the vehicle as overlapping the interested object when the driver is looking at the electronic display.

12. The system of claim 11, wherein the interested object and the occluding object are located outside the vehicle.

13. The system of claim 11, wherein the interested object and the occluding object are located at least substantially in front of the vehicle.

14. The system of claim 11, wherein the non-occluding graphic also does not overlay the interested object.

15. The system of claim 11, wherein the instructions when executed cause the system to track motion of the interested object using one or more vehicle sensors.

16. The system of claim 15, wherein the one or more vehicle sensors includes a camera that tracks the motion of the interested object.

17. The system of claim 16, wherein the camera tracks the motion of the interested object relative to the occluding object and the vehicle.

18. The system of claim 15, wherein the one or more vehicle sensors includes a range finder that tracks the motion of the interested object and one or more distances of the interested object relative to the occluding object and the vehicle.

19. The system of claim 15, wherein the vehicle sensors detect the interested object moving from a first position to a second position and the instructions when executed cause the system to reposition a location of the non-occluding graphic as displayed on the electronic display to correspond to the second position wherein the non-occluding graphic was previously displayed on the electronic display to correspond to the first position.

20. The system of claim 11, wherein the initial graphic is an outline that surrounds the interested object.