System and method for augmented reality head up display for vehicles

• US 10,281,721 B2
• Filed: 08/23/2017
• Issued: 05/07/2019
• Est. Priority Date: 08/23/2016
• Status: Active Grant

• Current Assignee: Vijyat Bhalla
• Original Assignee: 8696322 Canada, Inc.
• Inventors: Bhalla, Vijyat
• Primary Examiner(s): Lu, Shirley
• Application Number:
  US15/683,970
• Publication Number:
  US 20180059416A1
• Time in Patent Office: 622 Days
• CPC Class Codes: B60K 2360/176 Camera images
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/5899 Internet
• CPC Class Codes: B60K 2360/592 Data transfer involving external databases
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/22 Display screens
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/60 Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: B60W 2540/043 Identity of occupants
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/08 Interaction between the driver and the control system
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06F 16/24 : Querying
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/95 : structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/62 : Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: G08G 1/0137 : for specific applications
• CPC Class Codes: G08G 1/0175 : by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages

Abstract

What is disclosed is a method for augmented reality head up display in a user vehicle comprising: capturing, by input devices, input data; receiving, by a communications manager, said captured input data from said input devices; receiving, by said communications manager, data from a communications path from at least one of a data center, and one or more external parties; providing, by said communications manager, said received captured input data from said input devices, and said data received from said communications path to a central processing unit (CPU) coupled to said communications manager; generating, by said CPU, custom data based on at least one of said received captured input data, and said data received from said communications path; and displaying, by a windshield projector, said generated custom data on a windshield of said vehicle.

Claims

1. An augmented reality head up display (AR-HUD) system for a vehicle associated with a user, wherein said vehicle comprises a windshield, and said AR-HUD system comprises:

input devices comprising

a body camera worn by the user, and

one or more body-mounted audio recording devices worn by the user;

a windshield projector;

a communications path coupled to a data center and one or more external parties;

a central processing unit (CPU) communicatively coupled to

said input devices,

said communications path, and

said windshield projector

via a communications network manager;

further wherein

said input devices are used to capture input data comprising

a detected eye position,

one or more detected positions corresponding to one or more real world objects viewed through said windshield,

further wherein said one or more real world objects comprise a target vehicle,

an image of a target vehicle license plate associated with said target vehicle, and

data captured by the body camera or the one or more body mounted audio recording devices;

said communications network manager is used to

receive said captured input data from said input devices,

receive data from said communications path from at least one of said data center and said one or more external parties,

provide said received captured input data and said data received from said communications path to said CPU, further wherein

said CPU transmits one or more search requests based on said received captured input data, and

said data received from said communications path comprises one or more records retrieved based on said one or more search requests;

said CPU is used to generate custom data based on at least one of

said received captured input data, and

said data received from said communications path,

wherein said custom data comprises one or more augmented reality overlay elements, further wherein

 

said one or more augmented reality overlay elements are generated based on at least one of

 

said detected eye position,

 

said one or more detected positions corresponding to one or more real world objects viewed through said windshield,

 

said data captured by the body camera or the one or more body mounted audio recording devices, and

 

said one or more records retrieved based on said one or more search requests; and

said windshield projector is used to display said generated custom data on a windshield of the vehicle, wherein

at least one of said one or more augmented reality overlay elements is associated with said target vehicle and is displayed on said windshield proximate to one or more views of said target vehicle on said windshield.

2. The system of claim 1, wherein said one or more search requests comprise a first search request and a second search request;

said first search request is based on the image of the target vehicle license plate associated with said target vehicle; and

said second search request is based on the data captured by the body camera or the one or more body mounted audio recording devices.

3. The system of claim 1, wherein one of said one or more search requests is based on

the image of the target vehicle license plate associated with said target vehicle, and

the data captured by the body camera or the one or more body mounted audio recording devices.

4. The system of claim 1, wherein at least some of said data captured by the body camera or the one or more body mounted audio recording devices is related to said target vehicle.

5. The system of claim 1, wherein at least some part of said retrieved one or more records is transmitted to said user via an audio device or an earpiece carried by said user.

6. The system of claim 2, wherein

at least one of said one or more records is retrieved based on said second search request;

and

at least one of said one or more augmented reality overlay elements is generated based on

said detected eye position,

said one or more detected positions corresponding to one or more real world objects viewed through said windshield, and

said at least one of said one or more records retrieved based on said second search request.

7. The system of claim 2, wherein

at least some of said one or more records are retrieved based on said first and said second search request;

and

at least one of said one or more augmented reality overlay elements is generated based on

said detected eye position,

said one or more detected positions corresponding to one or more real world objects viewed through said windshield, and

said at least some of said one or more records retrieved based on said first and second search request.

8. The system of claim 5, wherein

at least one of said one or more records is retrieved based on said one of said one or more search requests;

and

at least one of said one or more augmented reality overlay elements is generated based on

said detected eye position,

said one or more detected positions corresponding to one or more real world objects viewed through said windshield, and

said at least one of said one or more records retrieved based on said one of said one or more search requests.

9. The system of claim 1, further wherein said one or more augmented reality overlay elements are generated based on said data captured by the body camera or the one or more body mounted audio recording devices.

10. The system of claim 9, wherein at least some part of said retrieved one or more records is transmitted to said user via an audio device or a earpiece carried by said user.

11. A method for augmented reality head up display in a user vehicle comprising:

capturing, by input devices, input data,

said input devices comprising

a body camera worn by the user, and

one or more body-mounted audio recording devices worn by the user, and said input data comprising

a detected eye position,

one or more detected positions corresponding to one or more real world objects viewed through said windshield,

further wherein said one or more real world objects comprise a target vehicle,

an image of a target vehicle license plate associated with said target vehicle, and

data captured by the body camera or the one or more body mounted audio recording devices;

receiving, by a communications network manager, said captured input data from said input devices;

providing, by said communications network manager, said received captured input data to a central processing unit (CPU) coupled to said communications network manager;

transmitting, by said CPU, one or more search requests based on said received captured input data;

receiving, by said communications manager, data from a communications path from at least one of

a data center, and

one or more external parties;

providing, by said communications manager, said data received from said communications path, to said CPU, wherein

said data received from said communications path comprises one or more records retrieved based on said one or more search requests;

generating, by said CPU, custom data based on at least one of

said received captured input data, and

said data received from said communications path,

wherein said custom data comprises one or more augmented reality overlay elements, further wherein

said one or more augmented reality overlay elements are generated based on at least one of

 

said detected eye position,

 

said one or more detected positions corresponding to one or more real world objects viewed through said windshield,

 

said data captured by the body camera or the one or more body mounted audio recording devices, and

 

said one or more records retrieved based on said one or more search requests; and

displaying, by a windshield projector, said generated custom data on a windshield of said vehicle, wherein

at least one of said one or more augmented reality overlay elements is associated with said target vehicle and is displayed on said windshield proximate to one or more views of said target vehicle on said windshield.

12. The method of claim 11, further wherein

said one or more real world objects comprise a first person of interest;

said input data comprises an image associated with said first person of interest;

said data received from said communications path comprises records corresponding to one or more persons of interest transmitted by said data center; and

further wherein at least one of said one or more augmented reality overlay elements is

associated with said first person of interest,

generated based on information obtained by said CPU cross referencing said image associated with said first person of interest with said records corresponding to one or more persons of interest, and

displayed on said windshield proximate to one or more views of said first person of interest.

13. The method of claim 12, further wherein

said one or more real world objects comprises a vehicle different to said target vehicle;

said input data comprises an image of a vehicle license plate associated with said different vehicle;

said data received from said communications path comprises records corresponding to one or more vehicles of interest transmitted by said data center,

wherein said transmission of said records corresponding to one or more vehicles of interest occurs without a search request being transmitted to said data center; and

further wherein at least one of said one or more augmented reality overlay elements is

associated with said different vehicle,

generated based on information obtained by said CPU cross referencing said image of said vehicle license plate associated with said different vehicle with said records corresponding to one or more vehicles of interest, and

displayed on said windshield proximate to one or more views of said different vehicle.

14. The method of claim 13, further wherein:

said one or more real world objects comprise a second person of interest;

said input data comprises an image of said second person of interest;

said transmitted one or more search requests comprises a first search request;

said first search request comprises said image of said second person of interest to said data center via said coupled communications network manager and said communications path;

said data received from said communications path by said communications network manager comprises one or more records retrieved based on said first search request; and

further wherein said at least one of said one or more augmented reality overlay elements are

associated with said second person of interest,

generated based on said retrieved one or more records, and

displayed on said windshield proximate to one or more views of said second person of interest on said windshield.

15. The system of claim 14, wherein at least two of

said at least one of said one or more augmented reality overlay elements associated with said target vehicle,

said at least one of said one or more augmented reality overlay elements associated with said first person of interest,

said at least one of said one or more augmented reality overlay elements associated with said different vehicle, and

said at least one of said one or more augmented reality overlay elements associated with said second person of interest,

are displayed on the windshield.

16. The method of claim 15, wherein said data received from said communications path comprises at least one of

an email;

an instant message;

a broadcast message; and

a telephone call.

17. The method of claim 16, wherein said custom data comprises one or more of

a notification associated with the email;

a messenger icon associated with the instant message;

an email icon associated with the email;

a broadcast message icon associated with the broadcast message;

a scrolling banner, and

a telephone icon associated with the telephone call.

18. The method of claim 17, wherein said input data comprises a speed corresponding to said target vehicle;

and

further wherein said at least one of said one or more augmented reality overlay elements comprise said speed.

19. The method of claim 18, further wherein

said at least one of said one or more augmented reality overlay elements comprises an indication that said speed exceeds a limit for a road travelled on by said target vehicle;

and

a ticket is generated based on said input data.

20. A method for an augmented reality head up display (AR-HUD) for a vehicle associated with a user, said vehicle having a windshield, said method comprising:

providing input devices comprising

a body camera worn by the user,

one or more body-mounted audio recording devices worn by the user, and

said input devices are used to capture input data comprising

a detected eye position,

one or more detected positions corresponding to one or more real world objects viewed through said windshield,

further wherein said one or more real world objects comprise a target vehicle,

an image of a target vehicle license plate associated with said target vehicle, and

data captured by the body camera or the one or more body mounted audio recording devices;

providing a communications network manager to

receive said captured input data from said input devices, and

receive data from a communications path coupled to at least one of a data center and one or more external parties;

providing a central processing unit (CPU), wherein

said communications network manager transmits said received captured input data and said data received from said communications path to said CPU,

said CPU transmits one or more search requests based on said received captured input data,

said data received from said communications path comprises one or more records retrieved based on said one or more search requests, and

said CPU generates custom data based on at least one of

said received captured input data, and

said data received from said communications path,

wherein said generated custom data comprises one or more augmented reality overlay elements, further wherein

 

said one or more augmented reality overlay elements are generated based on at least one of

 

said detected eye position,

 

said one or more detected positions corresponding to one or more real world objects viewed through said windshield,

 

said data captured by the body camera or the one or more body mounted audio recording devices, and

 

said one or more records retrieved based on said one or more search requests; and

providing a windshield projector to display said generated custom data on the windshield of the vehicle, wherein

at least one of said one or more augmented reality overlay elements is associated with said target vehicle and is displayed on said windshield proximate to one or more views of said target vehicle on said windshield.