DRIVER TRAINING SYSTEM USING HEADS-UP DISPLAY AUGMENTED REALITY GRAPHICS ELEMENTS

• US 20140272812A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Hing, Victor Ng-Thow, Beckwith, Lee
• Granted Patent: US 9,251,715 B2
• Time in Patent Office: Days
• US Class Current: 434/69
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G09B 9/04 for teaching control of land vehicles
• CPC Class Codes: G09B 9/05 the view from a vehicle being simulated G09B9/052, G09B9/058 take precedence

Abstract

A driver training system includes a training controller, a heads-up display device, and a driving cue adherence controller. The training controller is configured to receive inputs related to an operational state of a vehicle and an environment surrounding the vehicle, and to determine a driving cue based on the received inputs. The heads-up display device is configured to present the driving cue as an augmented reality graphic element in view of a driver by projecting graphic elements on a windshield of the vehicle. The driving cue adherence controller is configured to continuously determine a current level of adherence to the driving cue, and an aggregate level of adherence to the driving cue based on the continuously determined current level of adherence to the driving cue over a predetermined time period. The heads-up display device is configured to present the continuously determined aggregate level of adherence in view of the driver.

Claims

1. A driver training system, comprising:

a training controller configured to determine a driving cue based on inputs received from a control system of an associated vehicle related to an operational state of the associated vehicle and an environment surrounding the associated vehicle;

a heads-up display device configured to present the driving cue as an augmented reality graphic element in view of a driver of the associated vehicle by projecting graphic elements on a windshield of the associated vehicle.

2. The system according to claim 1, further comprising:

a driving cue adherence controller configured to continuously determine a current level of adherence to the driving cue presented to the driver.

3. The system according to claim 2, wherein the driving cue instructs a driving behavior and is at least one of:

a vehicle surrounding driving cue that instructs driving behavior to reduce a number of other vehicles in an immediate vicinity of the associated vehicle;

a merging driving cue that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway; and

a yielding driving cue that instructs driving behavior to yield a right of way at a particular path, and

the driving cue adherence controller communicates with the training controller and is configured to continuously determine the current level of adherence to the driving cue presented to the driver based on a variance between the driving behavior instructed by the driving cue and an actual driving behavior based on inputs received by the training controller.

4. The system according to claim 2, wherein

the driving cue adherence controller is configured to continuously determine an aggregate level of adherence to the driving cue based on the continuously determined current level of adherence to the driving cue over a predetermined time period, and

the heads-up display device is configured to present the continuously determined aggregate level of adherence to the driving cue in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

5. The system according to claim 4, wherein the continuously determined aggregate level of adherence to the driving cue is presented as an augmented reality graphic element separate from the driving cue.

6. The system according to claim 4, wherein the predetermined time period includes at least one of:

a time period beginning at a most recent power-on of the associated vehicle and continuing to a current time;

a time period beginning at a first power-on of the associated vehicle and continuing to the current time; and

a time period beginning at a resetting of the driver training system and continuing to the current time.

7. The system according to claim 2, wherein the driving cue is a vehicle surrounding grid that instructs driving behavior to reduce a number of vehicles in an immediate vicinity of the associated vehicle, the vehicle surrounding grid including a vehicle representative marker and a plurality of surrounding markers which at least partially surround the vehicle representative marker, the vehicle representative marker representing the associated vehicle and the plurality of surrounding markers each representing a unique region surrounding the associated vehicle,

the inputs received by the training controller from the associated vehicle control system indicate whether each unique region surrounding the associated vehicle is occupied by another vehicle based, and the training controller is configured to determine the vehicle surrounding grid such that each of the plurality of surrounding markers is highlighted when the unique region represented thereby is occupied by another vehicle,

the heads-up display device is configured to present the vehicle surrounding grid determined by the training controller as an augmented reality graphic element appearing on a sky focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle;

and

the driving cue adherence controller is configured to continuously determine the current level of adherence to the vehicle surrounding grid to increase as a percentage of the plurality of surrounding markers which are highlighted decreases.

8. The system according to claim 7, wherein

the driving cue adherence controller is configured to continuously determine an aggregate level of adherence to the vehicle surrounding grid based on the continuously determined current level of adherence to the vehicle surrounding grid over a predetermined time period, and

the heads-up display device is configured to present the continuously determined aggregate level of adherence to the vehicle surrounding grid as an augmented reality graphic element separate from the vehicle surrounding grid by projecting graphic elements on the windshield of the associated vehicle.

9. The system according to claim 2, wherein the driving cue is a merging guide that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway, the merging guide including a merging path line and a merging speed marker, the merging path line showing the desired merging path and the merging speed marker moving along the merging path line at the desired merging speed,

the inputs received by the training controller from the associated vehicle control system indicate the desired merging path and the desired merging speed, and the training controller is configured to determine the merging path line and the merging speed marker based on the desired merging path and desired merging speed indicated by the received inputs,

the heads-up display device is configured to present the merging path line and the merging speed marker determined by the training controller as contact-analog augmented reality graphic elements appearing on a road surface in view of the driver by projecting graphic elements on the windshield of the associated vehicle;

and

the driving cue adherence controller is configured to continuously determine the current level of adherence to the merging guide to increase as a positional variance of the associated vehicle from the merging path line and a speed variance of the associated vehicle from the merging speed marker decrease.

10. The system according to claim 9, wherein

the driving cue adherence controller is configured to continuously determine an aggregate level of adherence to the merging guide based on the continuously determined current level of adherence to the merging guide over a predetermined time period,

the heads-up display device is configured to present the continuously determined aggregate level of adherence to the merging guide as an augmented reality graphic element by projecting graphic elements on the windshield of the associated vehicle.

11. The system according to claim 2, wherein the driving cue is a yield confirmation marker that instructs driving behavior to yield a right of way at a particular path, and is presented to cover at least a portion of the path at which the associated vehicle is to yield,

inputs received by the training controller from the associated vehicle control system indicate a position and orientation of the yield confirmation marker in the environment surrounding the vehicle, and the training controller determines the yield confirmation marker to be at the position and orientation indicated by the received inputs,

the heads-up display device is configured to present the yield confirmation marker determined by the training controller as an augmented reality graphic element appearing in a frontal focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle, and

the driving cue adherence controller is configured to continuously determine the current level of adherence to the yield confirmation marker to increase as a speed of the associated vehicle decreases while the associated vehicle approaches the yield confirmation marker.

12. The system according to claim 11, wherein

the driving cue adherence controller is configured to continuously determine an aggregate level of adherence to the yield confirmation marker based on the current level of adherence to the yield confirmation marker over a predetermined time period,

the heads-up display device is configured to present the continuously determined aggregate level of adherence to the yield confirmation marker as an augmented reality graphic element separate from the yield confirmation marker by projecting graphic elements on the windshield of the associated vehicle.

13. The system according to claim 1, wherein

the training controller is configured to determine a plurality of driving cues, and the heads-up display device is configured to present the plurality of driving cues as augmented reality graphic elements in view of the driver of the associated vehicle,

one of the plurality of driving cues is a vehicle surrounding grid that instructs driving behavior to reduce a number of vehicles in an immediate vicinity of the associated vehicle, the vehicle surrounding grid including a vehicle representative marker and a plurality of surrounding markers which at least partially surround the vehicle representative marker, the vehicle representative marker representing the associated vehicle and the plurality of surrounding markers each representing a unique region surrounding the associated vehicle,

the inputs received by the training controller from the associated vehicle control system indicate whether each unique region surrounding the associated vehicle is occupied by another vehicle based, and the training controller is configured to determine the vehicle surrounding grid such that each of the plurality of surrounding markers is highlighted when the unique region represented thereby is occupied by another vehicle, and

the heads-up display device is configured to present the vehicle surrounding grid determined by the training controller as an augmented reality graphic element appearing on a sky focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

14. The system according to claim 1, wherein

the training controller is configured to determine a plurality of driving cues, and the heads-up display device is configured to present the plurality of driving cues as augmented reality graphic elements in view of the driver of the associated vehicle,

one of the plurality of driving cues is a merging guide that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway, the merging guide including a merging path line and a merging speed marker, the merging path line showing the desired merging path and the merging speed marker moving along the merging path line at the desired merging speed,

the inputs received by the training controller from the associated vehicle control system indicate the desired merging path and the desired merging speed, and the training controller is configured to determine the merging path line and the merging speed marker based on the desired merging path and desired merging speed indicated by the received inputs, and

the heads-up display device is configured to present the merging path line and the merging speed marker determined by the training controller as contact-analog augmented reality graphic elements appearing on a road surface in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

15. The system according to claim 1, wherein

the training controller is configured to determine a plurality of driving cues, and the heads-up display device is configured to present the plurality of driving cues as augmented reality graphic elements in view of the driver of the associated vehicle,

one of the plurality of driving cues is a yield confirmation marker that instructs driving behavior to yield a right of way at a particular path, and is presented to cover at least a portion of the path at which the associated vehicle is to yield,

inputs received by the training controller from the associated vehicle control system indicate a position and orientation of the yield confirmation marker in the environment surrounding the vehicle, and the training controller determines the yield confirmation marker to be at the position and orientation indicated by the received inputs, and

the heads-up display device is configured to present the yield confirmation marker determined by the training controller as an augmented reality graphic element appearing in a frontal focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

16. The system according to claim 13, wherein another of the plurality of driving cues is a merging guide that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway, the merging guide including a merging path line and a merging speed marker, the merging path line showing the desired merging path and the merging speed marker moving along the merging path line at the desired merging speed,

the inputs received by the training controller from the associated vehicle control system indicate the desired merging path and the desired merging speed, and the training controller is configured to determine the merging path line and the merging speed marker based on the desired merging path and desired merging speed indicated by the received inputs, and

the heads-up display device is configured to present the merging path line and the merging speed marker determined by the training controller as contact-analog augmented reality graphic elements appearing on a road surface in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

17. The system according to claim 16, wherein another of the plurality of driving cues is a yield confirmation marker that instructs driving behavior to yield a right of way at a particular path, and is presented to cover at least a portion of the path at which the associated vehicle is to yield,

inputs received by the training controller from the associated vehicle control system indicate a position and orientation of the yield confirmation marker in the environment surrounding the vehicle, and the training controller determines the yield confirmation marker to be at the position and orientation indicated by the received inputs, and

the heads-up display device is configured to present the yield confirmation marker determined by the training controller as an augmented reality graphic element appearing in a frontal focal plane in view of the driver by projecting graphic elements on the windshield of the associated vehicle.

18. A driving cue adherence controller for use with an associated driver training system in an associated vehicle, the driving cue adherence controller comprising at least one processor configured to:

receive inputs related to an operational state of the associated vehicle and an environment surrounding the associated vehicle;

receive inputs related to a driving cue presented to a driver of the associated vehicle as an augmented reality graphic element that instructs a driving behavior;

continuously determine a current level of adherence to the driving cue presented to the driver based on the received inputs of the operational state of the associated vehicle, the environment surrounding the associated vehicle, and the driving cue presented to the associated vehicle; and

continuously determine an aggregate level of adherence to the driving cue based on the continuously determined current level of adherence to the driving cue over a predetermined time period.

19. The driving cue adherence controller according to claim 18, wherein

the driving cue is at least one of:

a vehicle surrounding driving cue that instructs driving behavior to reduce a number of other vehicles in an immediate vicinity of the associated vehicle;

a merging driving cue that instructs driving behavior to follow a desired merging path and to travel at a desired merging speed when merging onto a roadway; and

a yielding driving cue that instructs driving behavior to yield a right of way at a particular path, and

the driving cue adherence controller is configured to continuously determine the current level of adherence to the driving cue based on a variance between driving behavior instructed by the driving cue and an actual driving behavior.

20. A driver training method, comprising:

presenting a driving cue which instructs a driving behavior to a driver as an augmented reality graphic element through a heads-up display device;

continuously determining a current level of adherence to the driving cue presented to the driver by monitoring a variance between the driving behavior instructed by the driving cue and an actual driving behavior;

continuously determining an aggregate level of adherence to the driving cue presented to the driver based on the current level of adherence to the driving cue determined over a predetermined time period; and

presenting the aggregate level of adherence to the driving cue to the driver as an augmented reality graphic element separate from the driving cue through the heads-up display device.