Ensuring occupant awareness in vehicles

• US 10,365,647 B2
• Filed: 03/29/2017
• Issued: 07/30/2019
• Est. Priority Date: 03/29/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Asghar, Ziad, Towal, Regan Blythe
• Primary Examiner(s): Wong, Yuen
• Application Number:
  US15/472,822
• Publication Number:
  US 20180284764A1
• Time in Patent Office: 853 Days
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/225 Direction of gaze
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 2540/24 Drug level, e.g. alcohol
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/0097 Predicting future conditions
• CPC Class Codes: B60W 50/082 Selecting or switching between different modes of propelling
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 50/16 Tactile feedback to the driver, e.g. vibration or force feedback to the driver on the steering wheel or the accelerator pedal
• CPC Class Codes: B60W 60/0053 from vehicle to occupant
• CPC Class Codes: B60W 60/0057 Estimation of the time available or required for the handover
• CPC Class Codes: B60W 60/0059 Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity
• CPC Class Codes: G01C 21/3676 Overview of the route on the road map

Abstract

In general, techniques are described for ensuring occupant awareness in vehicles. A device comprising a processor may be configured to perform the techniques. The processor may be configured to determine an event which will change an operating characteristic of the vehicle, and determine an extent of time remaining until an estimated occurrence of the expected event. The processor may also be configured to determine a process by which to inform the occupant of an operating context of the vehicle based on the determined extent of time remaining until the estimated occurrence the expected event, and perform the process by which to inform the occupant of the operating context of the vehicle. The device may also include a memory configured to store the process.

Claims

1. A method of providing information to ensure occupant awareness in a vehicle, the method comprising:

autonomously controlling, by the vehicle, operation of the vehicle;

determining, by one or more processors and concurrent with the vehicle autonomously controlling operation of the vehicle, an expected event is to occur, the expected event including transfer of control of the operation of the vehicle from the vehicle to an occupant;

determining, by the one or more processors and concurrent with the vehicle autonomously controlling operation of the vehicle, an extent of time remaining until an estimated occurrence of the expected event;

determining, by the one or more processors and concurrent with the vehicle autonomously controlling operation of the vehicle, a condition that triggers the expected event;

selecting, by the one or more processors and concurrent with the vehicle autonomously controlling operation of the vehicle, a process of a plurality of processes by which to inform the occupant of an operating context of the vehicle such that the process of the plurality of processes is capable of being performed within the determined extent of time, wherein a first subset of the plurality of processes are for higher levels of the determined extent of time compared to a second subset of the plurality of processes; and

performing, by the one or more processors and within the determined extent of time, the process of the plurality of processes by which to inform the occupant of the operating context of the vehicle.

2. The method of claim 1, further comprising determining a state of the occupant,

wherein selecting the process of the plurality of processes comprises selecting the process from the plurality of processes based on the determined extent of time remaining until the estimated occurrence of the expected event, the determined condition, and the state of the occupant.

3. The method of claim 2,

wherein determining the state of the occupant comprises determining a level of awareness of the occupant, and

wherein selecting the process of the plurality of processes comprises selecting the process from the plurality of processes based on the determined extent of time remaining until the estimated occurrence of the expected event, the determined condition, and the level of awareness of the occupant.

4. The method of claim 1,

wherein determining the condition comprises classifying the condition as being for communication via one or more audio processes of the plurality of processes, video processes of the plurality of process, and haptic processes of the plurality of processes, and

wherein selecting the process of the plurality of processes comprises selecting an audio process of the audio processes capable of apprising the occupant of the operating context within the determined extent of time remaining until the estimated occurrence of the expected event, a video process from the video processes capable of apprising the occupant of the operating context within the determined extent of time remaining until the estimated occurrence of the expected event, and a haptic process of the haptic process capable of apprising the occupant of the operating context within the determined extent of time remaining until the estimated occurrence of the expected event.

5. The method of claim 1, wherein selecting the process of the plurality of processes comprises:

selecting, based on the extent of time remaining until the estimated occurrence of the expected event and the determined condition, one of seconds-level processes capable of informing the occupant of the operating context within seconds, one of tens-of-seconds-level processes capable of informing the occupant of the operating context within tens of seconds, one of minutes-level processes capable of informing the occupant of the operating context within minutes, or one of tens-of-minutes-level process capable of informing the occupant of the operating context within tens of minutes; and

selecting the process of the plurality of processes from the selected one of the second-level processes, the tens-of-seconds-level process, the minutes-level processes, or the tens-of-minutes-level process.

6. The method of claim 1, further comprising:

determining whether the occupant will be able to take control of the vehicle when the extent of time remaining until the estimated occurrence of the transfer of the control of the operation of the vehicle to the occupant occurs; and

determining a level of alertness of the occupant, and

wherein selecting the process of the plurality of processes comprises determining the process based on the determined extent of time remaining until the estimated occurrence of the transfer of the control of the operation of the vehicle to the occupant, the determined condition, the determination of whether the occupant will be able to take control of the vehicle, and the determined level of alertness.

7. The method of claim 1, further comprising determining whether the occupant will be able to take control of the vehicle when the extent of time remaining until the estimated occurrence of the transfer of the control of the operation of the vehicle to the occupant occurs,

wherein selecting the process of the plurality of processes comprises determining, in response to determining that the occupant will not be able to take control of the vehicle, a default action to be performed autonomously by the vehicle, and

wherein performing the process comprises autonomously performing the default action.

8. The method of claim 7, wherein the default action comprises one of navigating to a side of a road on which the vehicle is traveling and stopping the vehicle, or stopping the vehicle.

9. The method of claim 1, wherein the process of the plurality of processes comprises displaying a destination summary of a trip for which the vehicle is currently autonomously controlling operation of the vehicle.

10. The method of claim 1, wherein the process of the plurality of processes comprises issuing one or more of an audio alert, issuing a haptic alert, entering a haptic co-piloting mode, and providing augmented reality assistance.

11. The method of claim 1, wherein the process of the plurality of processes comprises one or more of displaying a short or sped-up video summary of the operating context, providing a narrated summary of the operating context, displaying a short or sped-up first-person video of the operating context, utilizing an opaque display to highlight problems or show the operating context, and displaying a 360 degree image of the operating context.

12. The method of claim 1, wherein the process of the plurality of processes comprises one or more of presenting real-time and near-real-time video of the operating context.

13. The method of claim 1, wherein the process of the plurality of processes comprises one or more of displaying a map-based summary of a trip for which the vehicle is currently autonomously controlling operation of the vehicle, presenting an audio narrative of the operating context, and presenting a video narrative of the operating context is to be performed.

14. A device configured to provide information to ensure occupant awareness in a vehicle, the device comprising:

one or more processors configured to;

autonomously control operation of the vehicle;

determine, concurrent with the one or more processors autonomously controlling operation of the vehicle, an expected event is to occur, the expected vent including transfer of control of the operation of the vehicle form the vehicle to an occupant;

determine, concurrent with the one or more processors autonomously controlling operation of the vehicle, an extent of time remaining until an estimated occurrence of the expected event;

determine, concurrent with the one or more processors autonomously controlling operation of the vehicle, a condition that triggers the expected event;

determine, concurrent with the one or more processors autonomously controlling operation of the vehicle, a process of a plurality of processes by which to inform the occupant of an operating context of the vehicle such that the process of the plurality of processes is capable of being performed within the determined extent of time, wherein a first subset of the plurality of processes are for higher levels of the determined extent of compared to a second subset of the plurality of processes; and

perform, within the determined extent of time, the process of the plurality of processes by which to inform the occupant of the operating context of the vehicle; and

a memory configured to store the process.

15. The device of claim 14,

wherein the one or more processors are further configured to determine a state of the occupant, and

wherein the one or more processors are configured to select the process from the plurality of processes based on the determined extent of time remaining until the estimated occurrence of the expected event, the determined condition, and the state of the occupant.

16. The device of claim 14, wherein the one or more processors are configured to:

determine a level of awareness of the occupant, and

select the process from the plurality of processes based on the determined extent of time remaining until the estimated occurrence of the expected event, the determined condition, and the level of awareness of the occupant.

17. The device of claim 14, wherein the one or more processors are configured to:

classify the condition as being for communication via one or more classified audio processes of the plurality of processes, classified video processes of the plurality of process, and classified haptic processes of the plurality of processes, and

select an audio process of the audio process capable of apprising the occupant of the operating context within the determined extent of time remaining until the estimated occurrence of the expected event, a video process from the video processes capable of apprising the occupant of the operating context within the determined extent of time remaining until the estimated occurrence of the expected event, and a haptic process of the haptic process capable of apprising the occupant of the operating context within the determined extent of time remaining until the estimated occurrence of the expected event.

18. The device of claim 14, wherein the one or more processors are configured to:

select, based on the extent of time remaining until the estimated occurrence of the expected event and the determined condition, one of seconds-level processes capable of informing the occupant of the operating context within seconds, one of tens-of-seconds-level processes capable of informing the occupant of the operating context within tens of seconds, one of minutes-level processes capable of informing the occupant of the operating context within minutes, or one of tens-of-minutes-level process capable of informing the occupant of the operating context within tens of minutes; and

select the process of the plurality of processes from the selected one of the second-level processes, the tens-of-seconds-level process, the minutes-level processes, or the tens-of-minutes-level process.

19. The device of claim 14,

wherein the one or more processors are further configured to:

determine whether the occupant will be able to take control of the vehicle when the extent of time remaining until the estimated occurrence of the transfer of the control of the operation of the vehicle to the occupant occurs; and

determine a level of alertness of the occupant, and

wherein the one or more processors are configured to select the process of the plurality of processes based on the determined extent of time remaining until the estimated occurrence of the transfer of the control of the operation of the vehicle to the occupant, the determined condition, the determination of whether the occupant will be able to take control of the vehicle, and the determined level of alertness.

20. The device of claim 14,

wherein the one or more processors are further configured to determine whether the occupant will be able to take control of the vehicle when the extent of time remaining until the estimated occurrence of the transfer of the control of the operation of the vehicle to the occupant occurs,

wherein the one or more processors are configured to:

determine, in response to determining that the occupant will not be able to take control of the vehicle, a default action to be performed autonomously by the vehicle, and

autonomously perform the default action.

21. The device of claim 20, wherein the default action comprises one of navigating to a side of a road on which the vehicle is traveling and stopping the vehicle, or stopping the vehicle.

22. The device of claim 14, wherein the process of the plurality of processes comprises one of displaying a destination summary of a trip for which the vehicle is currently autonomously controlling operation of the vehicle, issuing an audio alert, issuing a haptic alert, entering a haptic co-piloting mode, providing augmented reality assistance, displaying a short or sped-up video summary of the operating context, providing a narrated summary of the operating context, displaying a short or sped-up first-person video of the operating context, utilizing an opaque display to highlight problems or show the operating context, displaying a 360 degree image of the operating context, presenting real-time or near-real-time video of the operating context, displaying a map-based summary of a trip for which the vehicle is currently autonomously controlling operation of the vehicle, presenting an audio narrative of the operating context, or presenting a video narrative of the operating context is to be performed.

23. A device configured to provide information to ensure occupant awareness in a vehicle, the device comprising:

means for autonomously controlling operation of the vehicle;

means for determining, concurrent with autonomous control of the operation of the vehicle, an expected event is to occur, the expected event including transfer of control of the operation of the vehicle from the vehicle to an occupant;

means for determining, concurrent with autonomous control of the operation of the vehicle, an extent of time remaining until an estimated occurrence of the expected event;

means for determining, concurrent with the vehicle autonomously controlling operation of the vehicle, a condition that triggers the expected event including transfer of the control of the operation of the vehicle to the occupant;

means for determining, concurrent with autonomous control of the operation of the vehicle, a process of a plurality of processes by which to inform the occupant of an operating context of the vehicle such that the process of the plurality of processes is capable of being performed within the determined extent of time, wherein a first subset of the plurality of processes are for higher levels of the determined extent of compared to a second subset of the plurality of processes; and

means for performing, within the determined extent of time, the process of the plurality of processes by which to inform the occupant of the operating context of the vehicle.

24. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to:

autonomously control operation of a vehicle;

determine, concurrent with the vehicle autonomously controlling operation of the vehicle, an expected event is to occur, the expected event including transfer of control of the operation of the vehicle from the vehicle to an occupant;

determine, concurrent with the vehicle autonomously controlling operation of the vehicle, a condition that triggers the expected event including transfer of the control of the operation of the vehicle to the occupant;

determine, concurrent with the vehicle autonomously controlling operation of the vehicle, a condition that triggers the expected event including transfer of the control of the operation of the vehicle to the occupant;

determine, concurrent with the vehicle autonomously controlling operation of the vehicle, a process of a plurality of processes by which to inform the occupant of an operating context of the vehicle such that the process of the plurality of processes is capable of being performed within the determined extent of time, wherein a first subset of the plurality of processes are for higher levels of the determined extent of compared to a second subset of the plurality of processes; and

perform, within the extent of time remaining until the estimated occurrence of the expected event, the process of the plurality of processes by which to inform the occupant of the operating context of the vehicle.