SYSTEMS, DEVICES, AND METHODS FOR CONTROLLING OPERATION OF WEARABLE DISPLAYS DURING VEHICLE OPERATION

• US 20200137665A1
• Filed: 10/23/2019
• Published: 04/30/2020
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: North Inc.
• Inventors: Judi, Terek, Klisz, Adrian, Reyes, Gabriel, Perry, Daniel
• Granted Patent: US 11,595,878 B2
• Time in Patent Office: Days
• CPC Class Codes: B60K 2360/149 Instrument input by detecting viewing direction not otherwise provided for
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/1868 according to driving situations
• CPC Class Codes: B60K 2360/195 Blocking or enabling display functions
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/29 Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04L 63/107 wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals
• CPC Class Codes: H04W 12/082 using revocation of authorisation
• CPC Class Codes: H04W 12/33 using wearable devices, e.g. using a smartwatch or smart-glasses
• CPC Class Codes: H04W 12/63 Location-dependent; Proximity-dependent
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/027 using movement velocity, acceleration information
• CPC Class Codes: H04W 4/40 for vehicles, e.g. vehicle-to-pedestrians [V2P]
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 48/04 : based on user or terminal location or mobility data, e.g. moving direction, speed

Abstract

The present systems, devices, and methods generally relate to controlling wearable displays during vehicle operation, and particularly to detecting when a user is operating a vehicle and restricting operation of a wearable display to prevent the user from being distracted. At least one processor of a wearable display system receives user context data from at least one user context sensor, and determines whether the user is operating a vehicle based on the user context data. If the user is operating a vehicle, presentation of at least one user interface is restricted. Unrestricted access can be restored by inputting an unlock input to override the restriction, or by analysis of additional user context data at a later time.

Claims

1. A system for presenting at least one user interface to a user of the system, the system comprising:

a wearable heads-up display (“

WHUD”

);

at least one processor;

at least one user context sensor communicatively coupled to the at least one processor; and

a non-transitory processor-readable medium communicatively coupled to the at least one processor, wherein the non-transitory processor-readable medium stores processor-executable instructions that, when executed by the at least one processor, cause the system to;

receive, by the at least one processor, user context data from the at least one user context sensor;

determine, by the at least one processor, that the user is operating a vehicle based on the user context data; and

after the at least one processor determines that the user is operating a vehicle, restrict, by the system, presentation of the at least one user interface to the user.

2. The system of claim 1 wherein the at least one user context sensor includes at least one microphone, the user context data includes captured audio data from the at least one microphone, and the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that a user is operating a vehicle based on the user context data cause the at least one processor to:

analyze at least one of a volume spectrum of the captured audio data, a volume pattern in the captured audio data, a frequency spectrum of the captured audio data, a frequency pattern in the captured audio data, or content of the captured audio data; and

determine that the user is operating a vehicle based on one or a combination of the volume spectrum of the captured audio data, a volume pattern in the captured audio data, a frequency spectrum of the captured audio data, a frequency pattern in the captured audio data, or content of the captured audio data.

3. The system of claim 1 wherein the at least one user context sensor includes at least one camera, the user context data includes captured visual data from the at least one camera, and the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user of the system is operating a vehicle based on the user context data cause the at least one processor to:

identify representations of vehicle features, vehicle accessories, or vehicle infrastructure in the captured visual data; and

determine that the user is operating a vehicle based on the identified vehicle features, vehicle accessories, or vehicle infrastructure.

4. The system of claim 1 wherein the at least one user context sensor includes at least one inertial measurement unit (“

IMU”

), the user context data includes captured user motion data from the at least one IMU, and the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user of the system is operating a vehicle based on the user context data cause the at least one processor to;

identify at least one of a velocity of the user, an acceleration of the user, and head motions made by the user based on the captured user motion data; and

determine that the user of the system is operating a vehicle based on one or a combination of at least one of a velocity of the user, an acceleration of the user, and head motions made by the user.

5. The system of claim 4 wherein the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user of the system is operating a vehicle based on one or a combination of at least one of a velocity of the user, an acceleration of the user, and head motions made by the user cause the at least one processor to perform at least one of:

determining that a velocity of the user exceeds a velocity threshold;

determining that an acceleration of the user exceeds an acceleration threshold; and

determining that the user'"'"'s head is frequently directed towards at least one of straight ahead, towards a rear-view mirror of a vehicle, towards a side-view mirror of a vehicle, or towards a side window of a vehicle.

6. The system of claim 1 wherein the at least one user context sensor includes a user eye-tracking system, the user context data includes captured user eye data from the user eye-tracking system, and the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user is operating a vehicle based on the user context data cause the at least one processor to:

analyze at least one of movement patterns of at least one eye of the user, gaze direction of the user, and a blink frequency of the user based on the captured user eye data; and

determine that a user of the system is operating a vehicle based on one or a combination of the at least one of movement patterns of at least one eye of the user, gaze direction of the user, and blink frequency of the user based on the captured user eye data.

7. The system of claim 1 wherein the at least one user context sensor includes at least one location sensor, the user context data includes captured user location data from the at least one location sensor, and the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user of the system is operating a vehicle based on the user context data cause the at least one processor to:

determine at least one of a velocity of the user or an acceleration of the user based on the captured user location data; and

determine that the user of the system is operating a vehicle based on one or a combination of at least one of a velocity of the user and an acceleration of the user.

8. The system of claim 1 wherein the user context data from the at least one user context sensor includes at least two of:

captured audio data from a microphone included in the at least one user context sensor;

captured visual data from a camera included in the at least one user context sensor;

captured user eye data from a user eye-tracking system included in the at least one user context sensor;

captured user motion data from an inertial measurement unit included in the at least one user context sensor; and

captured user location data from a location sensor included in the at least one user context sensor.

9. The system of claim 1 wherein the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user is operating a vehicle based on the user context data further cause the at least one processor to:

identify at least one recently used or currently used software application installed on the non-transitory processor-readable medium of the system;

determine that the at least one recently used or currently used software application is relevant to operating a vehicle.

10. The system of claim 1 wherein the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user is operating a vehicle based on the user context data cause the at least one processor to determine that the user is operating a vehicle including a car, van, bus, tractor, or truck, by causing the at least one processor to:

determine a seat of the vehicle occupied by the user based on the user context data;

determine a global location of the system based on location data from a location sensor included in the system or based on location settings stored in the non-transitory processor-readable medium;

determine an expected seat to be occupied by a driver of the vehicle based on the global location of the system; and

determine that the seat of the vehicle occupied by the user corresponds to the expected seat to be occupied by a driver of the vehicle.

11. The system of claim 1 wherein the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user is operating a vehicle based on the user context data cause the at least one processor to determine that the user is operating a vehicle including a bicycle or unicycle, by causing the at least one processor to:

receive user heart rate data or user breathing data from at least one user activity sensor; and

determine that a user has an elevated heart rate based on the user heart data or elevated breathing based on the user breathing data.

12. The system of claim 1 wherein the processor-executable instructions which when executed by the at least one processor cause the at least one processor to determine that the user is operating a vehicle based on the user context data cause the at least one processor to:

analyze the user context data to identify representations of features indicating that the user is not operating a vehicle; and

determine that a user is operating a vehicle if the user context data does not include representations of features indicating that the user is not operating a vehicle.

13. The system of claim 1 wherein the processor-executable instructions when executed by the at least one processor further cause the system to capture, by the at least one user context sensor, the user context data in response to the at least one processor receiving a user input.

14. The system of claim 1 wherein the processor-executable instructions which when executed by the at least one processor cause the system to restrict presentation of the at least one user interface to the user cause the system to completely disable presentation of the at least one user interface to the user.

15. The system of claim 1 wherein the processor-executable instructions which when executed by the at least one processor cause the system to restrict presentation of the at least one user interface to the user cause the system to disable presentation of at least one user interface which is not related to presenting at least one of:

a map, directional guidance, travel information, speed information, acceleration information, fuel consumption information, toll information, traffic information, weather information, or a clock display.

16. The system of claim 1 wherein the processor-executable instructions which when executed by the at least one processor cause the system to restrict presentation of the at least one user interface to the user cause the at least one processor to:

generate, by the at least one processor, a summary of content of the at least one user interface; and

present, by the system, the summary of content of the at least one user interface.

17. The system of claim 1 wherein the at least one user interface is a visual user interface, and wherein the processor-executable instructions which when executed by the at least one processor cause the system to restrict presentation of the at least one user interface to the user cause the system to modify a display size and/or a display position of the at least one user interface.

18. The system of claim 1 wherein the at least one user interface is a vibratory user interface, and wherein the processor-executable instructions which when executed by the at least one processor cause the system to restrict presentation of the at least one user interface to the user cause the system to reduce a vibration intensity and/or a vibration duration of the at least one user interface, or cause the system to modify a vibration pattern and/or a vibration frequency of the at least one user interface.

19. The system of claim 1 wherein the processor-executable instructions when executed by the at least one processor further cause the system to:

after the system restricts presentation of the at least one user interface to the user, receive, by the at least one processor, an unlock input from a user;

after the at least one processor receives the unlock input from a user, unrestrict, by the system, presentation of the at least one user interface to the user.

20. The system of claim 1 wherein the processor-executable instructions when executed by the at least one processor further cause the system to:

after the system restricts presentation of the at least one user interface to the user, receive, by the at least one processor, additional user context data from the at least one user context sensor;

determine, by the at least one processor, that the user is not operating a vehicle based on the additional user context data;

after the at least one processor determines that the user is no longer operating a vehicle, unrestrict, by the system, presentation of the at least one user interface to the user.