Systems and methods for mitigating vigilance decrement while maintaining readiness using augmented reality in a vehicle

• US 9,904,287 B1
• Filed: 05/04/2017
• Issued: 02/27/2018
• Est. Priority Date: 05/04/2017
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Research Institute, Inc. (Toyota Motor Corporation)
• Inventors: Kuffner, Jr., James J., Pratt, Gill A.
• Primary Examiner(s): Shafi, Muhammad
• Application Number:
  US15/586,347
• Time in Patent Office: 299 Days
• Field of Search: 701 23, 701400-541, 340988-996
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60R 1/22 for viewing an area outside the vehicle, e.g. the exterior of the vehicle
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/205 using a head-up display
• CPC Class Codes: B60R 2300/8086 for vehicle path indication
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/223 Posture, e.g. hand, foot, or seat position, turned or inclined
• CPC Class Codes: B60W 2540/225 Direction of gaze
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 2540/30 Driving style
• CPC Class Codes: B60W 2900/00 Indexing codes relating to the purpose of, or problem solved of road vehicle drive control systems not otherwise provided for in groups B60W30/00
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/08 Interaction between the driver and the control system
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

System, methods, and other embodiments described herein relate to mitigating vigilance decrement of a vehicle operator. In one embodiment, a method includes monitoring the operator by collecting operator state information using at least one sensor of the vehicle. The method includes computing an engagement level of the operator according to a vigilance model and the operator state information to characterize an extent of vigilance decrement presently experienced by the operator. The method includes rendering, on an augmented reality (AR) display, at least one graphical element as a function of the engagement level to induce the operator to maintain vigilance with respect to operation of the vehicle and a present operating environment around the vehicle.

Claims

1. A vigilance system for mitigating vigilance decrement of an operator in a vehicle, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing;

a monitoring module including instructions that when executed by the one or more processors cause the one or more processors to monitor the operator by collecting operator state information using at least one sensor of the vehicle;

a vigilance module including instructions that when executed by the one or more processors cause the one or more processors to compute an engagement level of the operator according to a vigilance model and the operator state information to characterize an extent of vigilance decrement presently experienced by the operator; and

a rendering module including instructions that when executed by the one or more processors cause the one or more processors to render, on an augmented reality (AR) display in the vehicle, at least one graphical element as a function of the engagement level by dynamically adjusting characteristics of how the at least one graphical element is rendered according to at least changes in the engagement level to induce vigilance within the operator with respect to autonomous operation of the vehicle and a present operating environment around the vehicle, and to facilitate preventing the operator from becoming disengaged from the autonomous operation and the present operating environment,

wherein the rendering module further includes instructions to render the at least one graphical element by generating the at least one graphical element from sensor data about the present operating environment that is external to the vehicle,

wherein the monitoring module, the vigilance module, and the rendering module execute on the one or more processors in parallel using a feedback loop to dynamically update the operator state information, to iteratively compute the engagement level and to adjust the rendering of the at least one graphical element according to changes observed in the engagement level,

wherein the rendering module further includes instructions to render the at least one graphical element by rendering the at least one graphical element with visual characteristics that are varied based, at least in part, on the changes in the engagement level over time, wherein the visual characteristics that are varied include at least an intensity of a graphical overlay of the present operating environment,

wherein the vehicle is operating according to at least a supervised autonomy standard while the monitoring module, the vigilance module, and the rendering module are executing in parallel, and wherein the rendering module further includes instructions to render the at least one graphical element to maintain readiness of the operator for when the vehicle performs a handover of manual control to the operator by providing visual cues to the operator through changing the at least one graphical element.

2. The vigilance system of claim 1, wherein the rendering module further includes instructions to render the at least one graphical element by rendering visualizations within the AR display using the sensor data about the present operating environment from one or more environmental sensors—

of the vehicle as the graphical overlay on the AR display,

wherein the rendering module further includes instructions to render the at least one graphical element by rendering internal state information of an autonomous driving module that indicates present decisions and situational awareness of the autonomous driving module about the present operating environment.

3. The vigilance system of claim 1, wherein the vigilance decrement indicates an extent to which the operator is aware of the present operating environment and autonomous operation of the vehicle through the present operating environment,

wherein the vigilance module further includes instructions to compute the engagement level by analyzing dynamic vehicle data along with the operator state information to account for the present operating environment of the vehicle in the engagement level, wherein the dynamic vehicle data includes at least telematics of the vehicle, a current location of the vehicle, and weather information, and wherein the engagement level indicates a present mental awareness of the operator in relation to the autonomous operation of the vehicle and the present operating environment.

4. The vigilance system of claim 1, wherein the monitoring module further includes instructions to monitor the operator by collecting at least gaze data from an eye-tracking camera that identifies a direction in which the operator is presently gazing, wherein the gaze data further includes an eye track that identifies a path of where the operator has been gazing over a period of time,

wherein the vigilance module further includes instructions to compute the engagement level by generating the engagement level as a statistical likelihood that the operator is engaged with the present operating environment, and wherein the vigilance model is a statistical model that characterizes a likely vigilance decrement of the operator as a function of the operator state information and the present operating environment.

5. The vigilance system of claim 1, wherein the vigilance module further includes instructions to update the vigilance model according to the operator state information and induced responses observed of the operator in response to the at least one graphical element to customize the rendering according to the operator.

6. The vigilance system of claim 1, wherein the rendering module further includes instructions to render the at least one graphical element by one or more of:

(i) varying an opacity of the graphical overlay within the AR display according to the changes in the engagement level,

(ii) varying a number of graphical elements displayed within the AR display according to the changes in the engagement level,

(iii) rendering the at least one graphical element within a line-of-sight of the operator in the AR display according to the changes in the engagement level, and

(v) rendering text within the AR display that includes questions for the operator about aspects of the present operating environment perceived by the at least one sensor of the vehicle when the engagement level indicates that the operator is experiencing vigilance decrement.

7. The vigilance system of claim 1, wherein the rendering module further includes instructions to render the at least one graphical element by rendering point cloud data from a LIDAR sensor of the vehicle as the graphic overlay within the AR display.

8. A non-transitory computer-readable medium for mitigating vigilance decrement of an operator in a vehicle and storing instructions that when executed by one or more processors cause the one or more processors to:

monitor the operator by collecting operator state information using at least one sensor of the vehicle;

compute an engagement level of the operator according to a vigilance model and the operator state information to characterize an extent of vigilance decrement presently experienced by the operator; and

render, on an augmented reality (AR) display in the vehicle, at least one graphical element as a function of the engagement level by dynamically adjusting characteristics of how the at least one graphical element is rendered according to at least changes in the engagement level to induce vigilance within the operator with respect to autonomous operation of the vehicle and a present operating environment around the vehicle, and to facilitate preventing the operator from becoming disengaged from the autonomous operation and the present operating environment,

wherein the instructions to render the at least one graphical element include instructions to generate the at least one graphical element from sensor data about the present operating environment that is external to the vehicle,

wherein the instructions to render the at least one graphical element include instructions to render the at least one graphical element with visual characteristics that are varied based, at least in part, on the changes in the engagement level over time, and as a graphical overlay of the present operating environment that is varied in intensity according to the changes in the engagement level,

wherein the instructions to monitor, compute, and render execute on the one or more processors in parallel using a feedback loop to dynamically update the operator state information, to iteratively compute the engagement level, and to adjust the rendering of the at least one graphical element according to the changes observed in the engagement level,

wherein the vehicle is operating according to at least a supervised autonomy standard while the instructions to monitor, compute, and render execute on the one or more processors in parallel, and wherein the instructions to render the at least one graphical element maintain readiness of the operator for when the vehicle performs a handover of manual control to the operator by providing visual cues to the operator through changing the at least one graphical element.

9. The non-transitory computer-readable medium of claim 8, wherein the instructions to monitor the operator further include instructions to collect at least gaze data from an eye-tracking camera that identifies a direction in which the operator is presently gazing, wherein the gaze data further includes an eye track that identifies a path of where the operator has been gazing over a period of time,

wherein the instructions to compute the engagement level further include instructions to generate the engagement level as a statistical likelihood that the operator is engaged with the present operating environment, and wherein the vigilance model is a statistical model that characterizes a likely vigilance decrement of the operator as a function of the operator state information and the present operating environment.

10. The non-transitory computer-readable medium of claim 8, wherein the the instructions to render the at least one graphical element include instructions to render visualizations within the AR display by adjusting the at least one graphical element according to the engagement level using data about the present operating environment from one or more environmental sensors of the vehicle as the graphical overlay on the AR display,

wherein the instructions to render the at least one graphical element include instructions to render internal state information of an autonomous driving module that indicates present decisions and situational awareness of the autonomous driving module about the present operating environment, and

wherein the autonomous driving module controls the vehicle according to the supervised autonomy standard that defines operation of the vehicle using at least supervised autonomy.

11. The non-transitory computer-readable medium of claim 8, wherein the instructions to render the at least one graphical element further include instructions to render point cloud data from a LIDAR sensor of the vehicle as the graphical overlay within the AR display.

12. A method of mitigating vigilance decrement of an operator in a vehicle with an augmented reality (AR) display, comprising:

monitoring the operator by collecting operator state information using at least one sensor of the vehicle;

computing an engagement level of the operator according to a vigilance model and the operator state information to characterize an extent of vigilance decrement presently experienced by the operator; and

rendering, on the AR display, at least one graphical element as a function of the engagement level by dynamically adjusting characteristics of how the at least one graphical element is rendered according to at least changes in the engagement level to induce vigilance within the operator with respect to autonomous operation of the vehicle and a present operating environment around the vehicle, and to facilitate preventing the operator from becoming disengaged from the autonomous operation and the present operating environment,

wherein rendering the at least one graphical element includes generating the at least one graphical element from sensor data about the present operating environment that is external to the vehicle,

wherein monitoring the operator, computing the engagement level, and rendering the at least one graphical element execute in parallel using a feedback loop of updated operator state information from the at least one sensor to iteratively compute the engagement level and adjust the rendering of the at least one graphical element according to the vigilance induced in the operator,

wherein rendering the at least one graphical element includes rendering the at least one graphical element with visual characteristics that are varied based, at least in part, on the changes in the engagement level over time, and as a graphical overlay of the present operating environment that is varied in intensity according to the changes in the engagement level,

wherein the vehicle is operating according to at least a supervised autonomy standard while the monitoring, computing, and rendering are executing in parallel, and wherein rendering the at least one graphical element maintains readiness of the operator for when the vehicle performs a handover of manual control to the operator by providing visual cues to the operator through changing the at least one graphical element.

13. The method of claim 12, wherein the vigilance decrement indicates an extent to which the operator is aware of the present operating environment and progress of the vehicle through the present operating environment,

wherein computing the engagement level further includes analyzing dynamic vehicle data along with the operator state information to account for the present operating environment of the vehicle when computing the engagement level, wherein the dynamic vehicle data includes at least telematics of the vehicle, a current location of the vehicle, and weather information and wherein the engagement level indicates a present mental awareness of the operator in relation to the autonomous operation of the vehicle and the present operating environment.

14. The method of claim 12, wherein monitoring the operator includes collecting at least gaze data from an eye-tracking camera that identifies a direction in which the operator is presently gazing, wherein the gaze data further includes an eye track that identifies a path of where the operator has been gazing over a period of time,

wherein computing the engagement level includes generating the engagement level as a statistical likelihood that the operator is engaged with the present operating environment, and wherein the vigilance model is a statistical model that characterizes a likely vigilance decrement of the operator as a function of the operator state information according to the present operating environment.

15. The method of claim 12, further comprising:

updating the vigilance model according to the operator state information and induced responses observed of the operator according to the rendering of the at least one graphical element to customize the rendering according to the operator.

16. The method of claim 12, wherein rendering the at least one graphical element further includes rendering visualizations within the AR display by adjusting the at least one graphical element according to the engagement level using data about the present operating environment from one or more environmental sensors of the vehicle as the graphical overlay on the AR display,

wherein rendering the at least one graphical element further includes rendering internal state information of an autonomous module that indicates present decisions and situational awareness of the autonomous module about the present operating environment, and

wherein the autonomous module controls the vehicle according to an autonomous driving standard for operating according to at least supervised autonomy.

17. The method of claim 12, wherein rendering the at least one graphical element as a function of the engagement level further includes one or more of:

(i) varying an opacity of an overlay within the AR display according to the changes of the engagement level,

(ii) varying a number of graphic elements displayed within the AR display according to the changes in the engagement level,

(iii) adding additional elements within a line-of-sight of the operator in the AR display according to the changes in the engagement level,

(iv) rendering point cloud data from a LiDAR sensor of the vehicle as the graphical overlay within the AR display when the engagement level indicates that the operator is experiencing vigilance decrement, and

(v) rendering text within the AR display that includes questions for the operator about aspects of the present operating environment perceived by the at least one sensor of the vehicle when the engagement level indicates that the operator is experiencing vigilance decrement.