Driver training system using heads-up display augmented reality graphics elements

• US 9,251,715 B2
• Filed: 03/15/2013
• Issued: 02/02/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Hing, Victor Ng-Thow, Beckwith, Lee
• Primary Examiner(s): Thai, Xuan
• Assistant Examiner(s): Bullington, Robert P
• Application Number:
  US13/832,209
• Publication Number:
  US 20140272812A1
• Time in Patent Office: 1,054 Days
• Field of Search: 434/69
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G09B 9/04 for teaching control of land vehicles
• CPC Class Codes: G09B 9/05 the view from a vehicle being simulated G09B9/052, G09B9/058 take precedence

Abstract

A driver training system includes a training controller, a heads-up display device, and a driving cue adherence controller. The training controller is configured to receive inputs related to an operational state of a vehicle and an environment surrounding the vehicle, and to determine a driving cue based on the received inputs. The heads-up display device is configured to present the driving cue as an augmented reality graphic element in view of a driver by projecting graphic elements on a windshield of the vehicle. The driving cue adherence controller is configured to continuously determine a current level of adherence to the driving cue, and an aggregate level of adherence to the driving cue based on the continuously determined current level of adherence to the driving cue over a predetermined time period. The heads-up display device is configured to present the continuously determined aggregate level of adherence in view of the driver.

Claims

1. A driver training system, comprising:

a training controller configured to determine a driving cue based on inputs received from a control system of an associated vehicle related to an operational state of the associated vehicle and an environment surrounding the associated vehicle, wherein the training controller receives inputs from at least one operational sensor and at least one environmental sensor and is implemented at least in part via a processor of a computer;

a driving cue adherence controller configured to continuously determine a current level of adherence to the driving cue presented to a driver of the associated; and

a heads-up display device configured to present the driving cue as an augmented reality graphic element in view of the driver of the associated vehicle by projecting graphic elements on a windshield of the associated vehicle,

wherein the driving cue instructs driving behavior based on positional weighting of other vehicles in an immediate vicinity of the associated vehicle.

2. The system according to claim 1, wherein the driving cue instructs a driving behavior and is at least one of:

a vehicle surrounding driving cue that instructs driving behavior to reduce a number of other vehicles in the immediate vicinity of the associated vehicle;

a merging driving cue that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway; and

a yielding driving cue that instructs driving behavior to yield a right of way at a particular path,

and the driving cue adherence controller communicates with the training controller and is configured to continuously determine the current level of adherence to the driving cue presented to the driver based on a variance between the driving behavior instructed by the driving cue and an actual driving behavior based on inputs received by the training controller.

3. The system according to claim 1, wherein

the driving cue adherence controller is configured to continuously determine an aggregate level of adherence to the driving cue based on the continuously determined current level of adherence to the driving cue over a predetermined time period;

and

the heads-up display device is configured to present the continuously determined aggregate level of adherence to the driving cue in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

4. The system according to claim 3, wherein the continuously determined aggregate level of adherence to the driving cue is presented as an augmented reality graphic element separate from the driving cue.

5. The system according to claim 3, wherein the predetermined time period includes at least one of:

a time period beginning at a most recent power-on of the associated vehicle and continuing to a current time;

a time period beginning at a first power-on of the associated vehicle and continuing to the current time; and

a time period beginning at a resetting of the driver training system and continuing to the current time.

6. The system according to claim 1, wherein the driving cue is a vehicle surrounding grid that instructs driving behavior to reduce a number of vehicles in the immediate vicinity of the associated vehicle, the vehicle surrounding grid including a vehicle representative marker and a plurality of surrounding markers which at least partially surround the vehicle representative marker, the vehicle representative marker representing the associated vehicle and the plurality of surrounding markers each representing a unique region surrounding the associated vehicle;

the inputs received by the training controller from the associated vehicle control system indicate whether each unique region surrounding the associated vehicle is occupied by another vehicle based, and the training controller is configured to determine the vehicle surrounding grid such that each of the plurality of surrounding markers is highlighted when the unique region represented thereby is occupied by another vehicle;

the heads-up display device is configured to present the vehicle surrounding grid determined by the training controller as an augmented reality graphic element appearing on a sky focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle; and

the driving cue adherence controller is configured to continuously determine the current level of adherence to the vehicle surrounding grid to increase as a percentage of the plurality of surrounding markers which are highlighted decreases.

7. The system according to claim 6, wherein

the driving cue adherence controller is configured to continuously determine an aggregate level of adherence to the vehicle surrounding grid based on the continuously determined current level of adherence to the vehicle surrounding grid over a predetermined time period, and

the heads-up display device is configured to present the continuously determined aggregate level of adherence to the vehicle surrounding grid as an augmented reality graphic element separate from the vehicle surrounding grid by projecting graphic elements on the windshield of the associated vehicle.

8. The system according to claim 1, wherein the driving cue is a merging guide that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway, the merging guide including a merging path line and a merging speed marker, the merging path line showing the desired merging path and the merging speed marker moving along the merging path line at the desired merging speed;

the inputs received by the training controller from the associated vehicle control system indicate the desired merging path and the desired merging speed, and the training controller is configured to determine the merging path line and the merging speed marker based on the desired merging path and desired merging speed indicated by the received inputs;

the heads-up display device is configured to present the merging path line and the merging speed marker determined by the training controller as contact-analog augmented reality graphic elements appearing on a road surface in view of the driver by projecting graphic elements on the windshield of the associated vehicle; and

the driving cue adherence controller is configured to continuously determine the current level of adherence to the merging guide to increase as a positional variance of the associated vehicle from the merging path line and a speed variance of the associated vehicle from the merging speed marker decrease.

9. The system according to claim 8, wherein the driving cue adherence controller is configured to continuously determine an aggregate level of adherence to the merging guide based on the continuously determined current level of adherence to the merging guide over a predetermined time period,

the heads-up display device is configured to present the continuously determined aggregate level of adherence to the merging guide as an augmented reality graphic element by projecting graphic elements on the windshield of the associated vehicle.

10. The system according to claim 1, wherein the driving cue is a yield confirmation marker that instructs driving behavior to yield a right of way at a particular path, and is presented to cover at least a portion of the path at which the associated vehicle is to yield;

inputs received by the training controller from the associated vehicle control system indicate a position and orientation of the yield confirmation marker in the environment surrounding the vehicle, and the training controller determines the yield confirmation marker to be at the position and orientation indicated by the received inputs;

the heads-up display device is configured to present the yield confirmation marker determined by the training controller as an augmented reality graphic element appearing in a frontal focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle;

and the driving cue adherence controller is configured to continuously determine the current level of adherence to the yield confirmation marker to increase as a speed of the associated vehicle decreases while the associated vehicle approaches the yield confirmation marker.

11. The system according to claim 10, wherein the driving cue adherence controller is configured to continuously determine an aggregate level of adherence to the yield confirmation marker based on the current level of adherence to the yield confirmation marker over a predetermined time period,

the heads-up display device is configured to present the continuously determined aggregate level of adherence to the yield confirmation marker as an augmented reality graphic element separate from the yield confirmation marker by projecting graphic elements on the windshield of the associated vehicle.

12. The system according to claim 1, wherein the training controller is configured to determine a plurality of driving cues, and the heads-up display device is configured to present the plurality of driving cues as augmented reality graphic elements in view of the driver of the associated vehicle,

one of the plurality of driving cues is a vehicle surrounding grid that instructs driving behavior to reduce a number of vehicles in the immediate vicinity of the associated vehicle, the vehicle surrounding grid including a vehicle representative marker and a plurality of surrounding markers which at least partially surround the vehicle representative marker, the vehicle representative marker representing the associated vehicle and the plurality of surrounding markers each representing a unique region surrounding the associated vehicle,

the inputs received by the training controller from the associated vehicle control system indicate whether each unique region surrounding the associated vehicle is occupied by another vehicle based, and the training controller is configured to determine the vehicle surrounding grid such that each of the plurality of surrounding markers is highlighted when the unique region represented thereby is occupied by another vehicle, and

the heads-up display device is configured to present the vehicle surrounding grid determined by the training controller as an augmented reality graphic element appearing on a sky focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

13. The system according to claim 1, wherein the training controller is configured to determine a plurality of driving cues, and the heads-up display device is configured to present the plurality of driving cues as augmented reality graphic elements in view of the driver of the associated vehicle,

one of the plurality of driving cues is a merging guide that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway, the merging guide including a merging path line and a merging speed marker, the merging path line showing the desired merging path and the merging speed marker moving along the merging path line at the desired merging speed,

the inputs received by the training controller from the associated vehicle control system indicate the desired merging path and the desired merging speed, and the training controller is configured to determine the merging path line and the merging speed marker based on the desired merging path and desired merging speed indicated by the received inputs, and

the heads-up display device is configured to present the merging path line and the merging speed marker determined by the training controller as contact-analog augmented reality graphic elements appearing on a road surface in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

14. The system according to claim 1, wherein

the training controller is configured to determine a plurality of driving cues, and the heads-up display device is configured to present the plurality of driving cues as augmented reality graphic elements in view of the driver of the associated vehicle,

one of the plurality of driving cues is a yield confirmation marker that instructs driving behavior to yield a right of way at a particular path, and is presented to cover at least a portion of the path at which the associated vehicle is to yield,

inputs received by the training controller from the associated vehicle control system indicate a position and orientation of the yield confirmation marker in the environment surrounding the vehicle, and the training controller determines the yield confirmation marker to be at the position and orientation indicated by the received inputs, and

the heads-up display device is configured to present the yield confirmation marker determined by the training controller as an augmented reality graphic element appearing in a frontal focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

15. The system according to claim 12, wherein another of the plurality of driving cues is a merging guide that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway, the merging guide including a merging path line and a merging speed marker, the merging path line showing the desired merging path and the merging speed marker moving along the merging path line at the desired merging speed,

the inputs received by the training controller from the associated vehicle control system indicate the desired merging path and the desired merging speed, and the training controller is configured to determine the merging path line and the merging speed marker based on the desired merging path and desired merging speed indicated by the received inputs, and

the heads-up display device is configured to present the merging path line and the merging speed marker determined by the training controller as contact-analog augmented reality graphic elements appearing on a road surface in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

16. The system according to claim 15, wherein another of the plurality of driving cues is a yield confirmation marker that instructs driving behavior to yield a right of way at a particular path, and is presented to cover at least a portion of the path at which the associated vehicle is to yield,

inputs received by the training controller from the associated vehicle control system indicate a position and orientation of the yield confirmation marker in the environment surrounding the vehicle, and the training controller determines the yield confirmation marker to be at the position and orientation indicated by the received inputs, and

the heads-up display device is configured to present the yield confirmation marker determined by the training controller as an augmented reality graphic element appearing in a frontal focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

17. A driving cue adherence controller for use with an associated driver training system in an associated vehicle, the driving cue adherence controller comprising at least one operational sensor and at least one environmental sensor and at least one processor of a computer which is configured to:

receive inputs from the at least one operational sensor related to an operational state of the associated vehicle and an environment surrounding the associated vehicle;

receive inputs related to a driving cue presented to a driver of the associated vehicle as an augmented reality graphic element that instructs a driving behavior, wherein the driving cue instructs driving behavior based on positional weighting of other vehicles in an immediate vicinity of the associated vehicle;

continuously determine a current level of adherence to the driving cue presented to the driver based on the received inputs from the at least one operational sensor of the operational state of the associated vehicle, from the at least one environmental sensor of the environment surrounding the associated vehicle, and the driving cue presented to the associated vehicle; and

continuously determine an aggregate level of adherence to the driving cue based on the continuously determined current level of adherence to the driving cue over a predetermined time period.

18. The driving cue adherence controller according to claim 17, wherein the driving cue is at least one of:

a vehicle surrounding driving cue that instructs driving behavior to reduce a number of other vehicles in the immediate vicinity of the associated vehicle;

a merging driving cue that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway; and

a yielding driving cue that instructs driving behavior to yield a right of way at a particular path, and the driving cue adherence controller is configured to continuously determine the current level of adherence to the driving cue based on a variance between driving behavior instructed by the driving cue and an actual driving behavior.

19. A driver training method, comprising:

presenting a driving cue which instructs a driving behavior to a driver as an augmented reality graphic element through a heads-up display device, wherein the driving cue instructs driving behavior based on positional weighting of other vehicles in an immediate vicinity of the associated vehicle such that other vehicles in a same lane as the associated vehicle are weighted more than other vehicles in a different lane than the associated vehicle;

continuously determining a current level of adherence to the driving cue presented to the driver by monitoring a variance between the driving behavior instructed by the driving cue and an actual driving behavior;

continuously determining an aggregate level of adherence to the driving cue presented to the driver based on the current level of adherence to the driving cue determined over a predetermined time period; and

presenting the aggregate level of adherence to the driving cue to the driver as an augmented reality graphic element separate from the driving cue through the heads-up display device,

wherein at least one of the presenting the driving cue, determining the current level, determining the aggregate level, or presenting the aggregate level is based on inputs received from at least one operational sensor and at least one environmental sensor and is implemented via a processor of a computer.