Wireless data sharing between a mobile client device and a three-dimensional heads-up display unit

• US 10,232,710 B2
• Filed: 03/24/2016
• Issued: 03/19/2019
• Est. Priority Date: 03/24/2016
• Status: Expired due to Fees

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Sisbot, Emrah Akin
• Primary Examiner(s): Okebato, Sahlu
• Application Number:
  US15/080,433
• Publication Number:
  US 20170274771A1
• Time in Patent Office: 1,090 Days
• CPC Class Codes: B60K 2360/165 Videos and animations
• CPC Class Codes: B60K 2360/184 Displaying the same information on different displays
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/334 Projection means
• CPC Class Codes: B60K 2360/344 for additionally illuminating mechanical elements, e.g. pointers or control knobs
• CPC Class Codes: B60K 2360/566 Mobile devices displaying vehicle information
• CPC Class Codes: B60K 2360/589 Wireless data transfers
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/211 producing three-dimensional [3D] effects, e.g. stereoscopic images
• CPC Class Codes: B60K 35/22 Display screens
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/29 Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

The disclosure includes a system and method for wireless data sharing between a mobile client device and a three-dimensional heads-up display unit. A system may include a three-dimensional heads-up display unit (“3D HUD”) installed in a vehicle. The system may include a memory storing instructions that, when executed, cause the system to: establish a peer-to-peer video stream between a mobile client device and the vehicle; generate live video data for providing a video stream for causing a screen of the mobile client device to display visual content of the 3D HUD that includes substantially live images depicting what the driver of the vehicle sees when looking at the 3D HUD; and stream the live video data to the mobile client device to cause the screen of the mobile client device to display the video stream that depicts what the driver of the vehicle sees when looking at the 3D HUD.

Claims

1. A computer-implemented method comprising:

pairing a three-dimensional heads-up display unit (“

3D HUD”

) installed in a vehicle with a mobile client device, wherein the pairing is established via a wireless communicative coupling shared by the vehicle and the mobile client device;

generating live video data for causing a screen of the mobile client device to display visual content that is viewable by a driver of the vehicle when looking at the 3D HUD;

streaming the live video data to the mobile client device via the wireless communicative coupling shared by the vehicle and the mobile client device so that the screen of the mobile client device displays the visual content that depicts in at least substantially real time what the driver of the vehicle sees when looking at the 3D HUD;

receiving mobile client device location data describing a set of global positioning system coordinates for the mobile client device; and

displaying, on the 3D HUD, an electronic map that depicts a mobile client device location and a graphical overlay depicting a real-life image or an avatar of a user associated with the mobile client device.

2. The method of claim 1, wherein the pairing includes establishing an encrypted peer-to-peer video stream between the 3D HUD of the vehicle and the mobile client device so that the mobile client device streams the live video data via the encrypted peer-to-peer video stream.

3. The method of claim 1, wherein the communicative coupling is encrypted so that only the mobile client device may stream the live video data.

4. The method of claim 1, wherein the wireless communicative coupling includes an encrypted virtual private network tunnel that is only accessible by the vehicle and the mobile client device.

5. The method of claim 1, wherein the live video data includes one or more real world images captured by a camera mounted to the vehicle that captures the one or more real world images as viewed by the driver when looking at the 3D HUD.

6. The method of claim 5, wherein the live video data includes graphical data that describes the graphical overlay that is viewable by the driver when looking at a real world through the 3D HUD so that the screen of the mobile client device displays (1) the one or more real world images that the driver sees when looking at the 3D HUD and (2) the graphical overlay that the driver sees when looking at the 3D HUD.

7. The method of claim 1, further comprising:

determining vehicle location data describing a set of global positioning system coordinates for the vehicle; and

transmitting the vehicle location data to the mobile client device to cause the screen of the mobile client device to display a graphical user interface map depicting a vehicle location on the graphical user interface map.

8. The method of claim 1, wherein the graphical overlay includes the real-life image or the avatar as indicated by a user profile associated with the user.

9. The method of claim 1, further comprising:

determining that the vehicle is located close enough to the user of the mobile client device so that the user is viewable by the driver when looking through the 3D HUD;

generating graphical data for displaying an element to distinguish the user among other items viewable by the driver through the 3D HUD; and

updating the graphical overlay in the 3D HUD with the element on top of the user so that the driver can identify the user among the other items viewable by the driver through the 3D HUD.

21. The method of claim 1, wherein the 3D HUD is operable so that (1) the mobile client device location is viewable by the driver on at least a portion of the 3D HUD and (2) the driver does not need to adjust eye focus in order to view the electronic map on the 3D HUD.

10. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:

establish a peer-to-peer video stream between an information sharing application associated with a three-dimensional heads-up display unit (“

3D HUD”

) installed in a vehicle and a mobile application of a mobile client device via a wireless communicative coupling shared by the vehicle and the mobile client device;

generate live video data for causing a screen of the mobile client device to display visual content that includes images that are viewable by a driver of the vehicle when looking at the 3D HUD;

stream the live video data to the mobile client device via the wireless communicative coupling shared by the vehicle and the mobile client device, wherein the mobile application and the live video data are executed by a processor of the mobile client device to cause the screen of the mobile client device to display the peer-to-peer video stream that includes the visual content that includes the images that are viewable by the driver of the vehicle when looking at the 3D HUD;

receive mobile device location data describing a set of global positioning system coordinates for the mobile client device; and

display, on the 3D HUD, an electronic map that depicts a mobile client device location and a graphical overlay depicting a real-life image or an avatar of a user associated with the mobile client device.

11. The computer program product of claim 10, wherein the visual content is streamed to the mobile client device in substantially real time.

12. The computer program product of claim 10, wherein the computer-readable program when executed on a computer is further configured to cause the computer to:

determine vehicle location data describing a set of global positioning system coordinates for the vehicle; and

transmit the vehicle location data to the mobile client device, wherein the processor of the mobile client device executes the mobile application and the vehicle location data to cause the screen of the mobile client device to display a graphical user interface map depicting a location of the vehicle on the graphical user interface map and an estimated time of arrival for the vehicle to arrive at the location of the mobile client device.

13. The computer program product of claim 10, wherein the graphical overlay further includes turn-by-turn navigation instructions for the vehicle to reach the mobile client device location.

14. The computer program product of claim 10, wherein the computer-readable program when executed on a computer is further configured to cause the computer to:

determine that the vehicle is located close enough to the user of the mobile client device so that the user is viewable by the driver when looking through the 3D HUD;

generate graphical data for displaying an element to distinguish the user among other items viewable by the driver through the 3D HUD; and

update the graphical overlay in the 3D HUD with the element on top of the user so that the driver can identify the user among the other items viewable by the driver through the 3D HUD.

15. The computer program product of claim 10, wherein the computer-readable program when executed on a computer is further configured to cause the computer to:

determine a reference frame that includes a portion of the 3D HUD the driver of the vehicle is viewing; and

convert the set of global positioning system coordinates for the mobile client device to Cartesian coordinates that are included in the reference frame;

wherein the electronic map is displayed in the reference frame.

16. A system comprising:

a three-dimensional heads-up display unit (“

3D HUD”

) installed in a vehicle; and

a memory storing instructions that, when executed, cause the system to;

establish a peer-to-peer video stream between a mobile client device and the vehicle via a wireless communicative coupling shared by the vehicle and the mobile client device;

generate live video data for causing a screen of the mobile client device to display substantially live images that depict what a driver of the vehicle sees when looking at the 3D HUD;

stream the live video data to the mobile client device via the wireless communicative coupling shared by the vehicle and the mobile client device to cause the screen of the mobile client device to display the peer-to-peer video stream that depicts in substantially real time what the driver of the vehicle sees when looking at the 3D HUD; and

receive mobile client device location data describing a set of global positioning system coordinates for the mobile client device;

wherein the 3D HUD displays an electronic map that depicts a mobile client device location and a graphical overlay depicting a real-life image or an avatar of a user associated with the mobile client device.

17. The system of claim 16, wherein the instructions when executed cause the system to also:

determine vehicle location data describing a set of global positioning system coordinates for the vehicle; and

transmit the vehicle location data to the mobile client device to cause the screen of the mobile client device to display to a graphical user interface map depicting a vehicle location on the graphical user interface map.

18. The system of claim 16, wherein the graphical overlay includes the real-life image or the avatar as indicated by a user profile associated with the user.

19. The system of claim 16, wherein the instructions when executed cause the system to also:

determine that the vehicle is located close enough to the user of the mobile client device so that the user is viewable by the driver when looking through the 3D HUD;

generate graphical data for displaying an element to distinguish the user among other items viewable by the driver through the 3D HUD; and

update the graphical overlay in the 3D HUD with the element proximate to the user so that the driver can identify the user among the other items viewable by the driver through the 3D HUD.

20. The system of claim 16, wherein the instructions when executed cause the system to also:

determine a reference frame that includes a portion of the 3D HUD the driver of the vehicle is viewing;

determine other content displayed in the 3D HUD; and

convert the set of global positioning system coordinates for the mobile client device to Cartesian coordinates that are included in the reference frame; and

wherein the electronic map is displayed in the reference frame.