Three dimensional heads-up display unit including visual context for voice commands

• US 10,140,770 B2
• Filed: 03/24/2016
• Issued: 11/27/2018
• Est. Priority Date: 03/24/2016
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Sisbot, Emrah Akin
• Primary Examiner(s): Hajnik, Daniel F
• Application Number:
  US15/080,394
• Publication Number:
  US 20170278305A1
• Time in Patent Office: 978 Days
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 30/00 Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images in microscopes G02B21/22
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

The disclosure includes a system and method for wireless data sharing between a mobile client device and a three-dimensional heads-up display unit. A system may include a three-dimensional heads-up display unit (“3D HUD”) installed in a vehicle. The system may include a memory storing instructions that, when executed, cause the system to: establish a peer-to-peer video stream between a mobile client device and the vehicle; generate live video data for providing a video stream for causing a screen of the mobile client device to display visual content of the 3D HUD that includes substantially live images depicting what the driver of the vehicle sees when looking at the 3D HUD; and stream the live video data to the mobile client device to cause the screen of the mobile client device to display the video stream that depicts what the driver of the vehicle sees when looking at the 3D HUD.

Claims

1. A computer-implemented method comprising:

capturing an image of an object in a vehicle environment, wherein the image includes a scene in the vehicle environment that is viewable by a driver when looking through a three dimensional heads-up display unit (“

3D HUD”

) included in a vehicle that is driven by the driver;

receiving GPS data describing a geographic location of the vehicle;

receiving directory data describing one or more destinations proximate to the geographic location of the vehicle and operation information for the one or more destinations;

determining an identity of the object, sensor coordinates of the object, and a rule indicated by the object based at least in part on a set of object priors including images of the one or more destinations;

determining time data describing a current time;

determining a relevance of the object to the driver based on the time data, the rule, and the operation information, wherein the current time and the rule indicate that the object is relevant to the driver;

selecting a voice command that is relevant to the object and the driver based on the identity of the object and the relevance of the object to the driver, wherein upon receipt of the voice command from the driver, a voice command system controls one or more vehicle functions that correspond to the voice command;

identifying three dimensional overlay data (“

3D overlay data”

) associated with the object based on the relevance of the object to the driver, wherein the 3D overlay data describes the voice command; and

painting a 3D overlay described by the 3D overlay data at a first location in the 3D HUD by transforming the sensor coordinates of the object to point of view coordinates of the driver which are operable to cause the 3D overlay to be painted on the 3D HUD in a proximity to the object as viewed by the driver through the 3D HUD so that the driver can discern from the proximity which voice commands are available for the object.

2. A computer-implemented method comprising:

capturing an image of an object in a vehicle environment, wherein the image includes a scene in the vehicle environment that is viewable by a driver when looking through a through a 3D HUD included in a vehicle that is driven by the driver;

receiving GPS data describing a geographic location of the vehicle;

receiving directory data describing one or more destinations proximate to the geographic location of the vehicle and operation information for the one or more destinations;

determining an identity of the object, sensor coordinates of the object, and a rule indicated by the object;

determining time data describing a current time;

determining that the object is relevant to the driver based on the current time and the rule;

selecting a voice command that is relevant to the object based on the identity of the object, wherein upon receipt of the voice command from the driver, a voice command system controls one or more vehicle functions that correspond to the voice command;

identifying 3D overlay data associated with the object that describes the voice command; and

painting a 3D overlay described by the 3D overlay data at a first location in the 3D HUD by transforming the sensor coordinates of the object to point of view coordinates of the driver which are operable to cause the 3D overlay to be painted on the 3D HUD in a proximity to the object as viewed by the driver through the 3D HUD so that the driver can discern from the proximity which voice commands are available for the object, wherein the 3D overlay describes the voice command that is available to the driver and wherein the driver does not need to adjust eye focus in order to view the 3D overlay data.

3. The method of claim 2, wherein the identity of the object is determined based at least in part on a set of object priors including images of the one or more destinations.

4. The method of claim 2, wherein the object is a parking sign and the rule describes times of day when it is legal to park in a parking space.

5. The method of claim 2, wherein the object is a business and the operation information describes hours of operation for the business.

6. The method of claim 5, wherein the operation information and the time data indicate that the one or more destinations are available at the current time.

7. The method of claim 6, wherein the selected voice command includes requesting information about the business and the 3D overlay includes a graphic indicating that the driver may request information about the business.

8. The method of claim 5, further comprising:

retrieving social data for the driver;

wherein determining that the object is relevant is further based on the social data.

9. The method of claim 8, wherein the selected voice command includes calling the business and the 3D overlay includes a graphic indicating that the driver may request that a phone call be made to the business.

10. The method of claim 8, wherein the selected voice command includes requesting information about the business and the 3D overlay includes a graphic indicating that the driver may request information about the business.

11. The method of claim 2, further comprising determining weather data describing a weather forecast for the geographic location.

12. The method of claim 11, wherein the object is an outdoor facility.

13. The method of claim 12, wherein the weather forecast indicates weather that is inconsistent with the operation information for the outdoor facility.

14. The method of claim 13, wherein the selected voice command includes requesting information about the outdoor facility and the 3D overlay includes a graphic indicating that the driver may request information about the outdoor facility.

15. The method of claim 12, wherein the weather forecast indicates weather that is consistent with the operation information.

16. The method of claim 15, the selected voice command includes requesting the weather forecast and the 3D overlay includes a graphic indicating that the driver may request the weather forecast.

17. The method of claim 12, wherein the outdoor facility is a park.

18. The method of claim 17, wherein the operation information describes one or more rules for the park, the selected voice command includes requesting the rules for the park and the 3D overlay includes a graphic indicating that the driver may request the rules for the park.

19. The method of claim 2, wherein the object is a parking space, the selected voice command includes requesting the vehicle automatically park in the parking space and the 3D overlay includes a graphic indicating that the driver may request that the vehicle automatically park in the parking space.

20. A computer-implemented method comprising:

capturing an image of a parking space and a parking sign associated with the parking space, wherein the parking space is viewable by a driver when looking through a through a 3D HUD included in a vehicle that is driven by the driver;

determining sensor coordinates of the parking sign and an identity of the parking sign based at least in part on a set of object priors including images a set of parking signs;

determining a present time;

determining a rule associated with the parking sign based on the identity of the parking sign, wherein applicability of the rule varies based on the present time;

determining whether the vehicle can legally park in the parking space based on the rule associated with the parking sign and the present time;

selecting a voice command that is relevant to the parking space based on the rule associated with the parking sign, wherein upon receipt of the voice command from the driver, a voice command system controls one or more vehicle functions that correspond to the voice command;

determining 3D overlay data describing whether the vehicle can legally park in the parking space, wherein the 3D overlay data describes the voice command; and

painting a 3D overlay described by the 3D overlay data at a first location in the 3D HUD by transforming the sensor coordinates of the parking sign to point of view coordinates of the driver which are operable to cause the 3D overlay to be painted on the 3D HUD in a proximity to the parking sign as viewed by the driver through the 3D HUD so that the driver can discern from the proximity which voice commands are available for the parking sign.

21. The method of claim 20, wherein the voice command is for engaging an automatic parking system of the vehicle to cause the vehicle to automatically park in the parking space.