Impaired operator detection and interlock apparatus

• US 10,559,307 B1
• Filed: 02/13/2019
• Issued: 02/11/2020
• Est. Priority Date: 02/13/2019
• Status: Active Grant

• Current Assignee: The Notebook, LLC
• Original Assignee: The Notebook, LLC
• Inventors: Khaleghi, Karen Elaine
• Primary Examiner(s): Abebe, Daniel
• Application Number:
  US16/275,132
• Time in Patent Office: 363 Days
• CPC Class Codes: B60K 28/06 responsive to incapacity of driver
• CPC Class Codes: B60K 28/063 preventing starting of vehicles
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/66 for extracting parameters related to health condition detecting or measuring for diagnostic purposes A61B5/00
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

Systems and methods are disclosed configured to detect impairment issues, and via an interlock device, inhibit operation of an item of equipment when impairment is detected. The interlock device may comprise a solid state relay, an electromechanical relay, and/or a solenoid. The interlock device may perform power isolation and/or may use a mechanism, such as a rotating cam or gear, to immobilize a control and/or other components. Based on detected impairment, a determination is made as to whether the interlock is to be activated or deactivated.

Claims

1. An electronic interlock system configured to selectively immobilize equipment using an interlock device, comprising:

a network interface;

at least one computing device;

computer readable memory including instructions operable to be executed by the at least one computing device to perform a set of actions, configuring the at least one computing device to;

receive in real time, over a network via the network interface, a digitized human vocal expression of a first user and one or more digital images of the first user from a remote device;

process the received digitized human vocal expression to;

generate a power spectrum of the received digitized human vocal expression;

use the generated power spectrum to identify quiet periods in the digitized human vocal expression;

identify, using the identified quiet periods, length of pauses between words in the digitized human vocal expression;

convert audible speech in the human vocal expression to text and characterize audible speech in the human vocal expression using the text;

compare the determined characteristics of the human vocal expression with baseline, historical characteristics of human vocal expressions associated with the first user to identify changes in human vocal expression characteristics of the first user;

process the received one or more images to detect characteristics of the first user face, including detecting if one or more of the following are present;

a sagging lip, facial droop, and dilated pupils;

compare one or more of the detected characteristics of the first user face with baseline, historical characteristics of the first user face accessed from a data store, and identify changes in characteristics of the first user face;

based at least on part on;

identified changes in human vocal expression characteristics of the first user,

the identified length of pauses between words in the digitized human vocal expression, and

identified changes in characteristics of the first user face,

determine an impairment categorization for the first user; and

use the impairment categorization, determined based at least in part on the identified changes in human vocal expression characteristics of the first user, the identified length of pauses between words in the digitized human vocal expression, and the identified changes in characteristics of the first user face, for the first user to determine whether the equipment is to be immobilized using the interlock device;

at least partly in response to a determination that the equipment is to be immobilized, controlling the interlock device so as to immobilize the equipment.

2. The electronic interlock system as defined in claim 1, wherein the interlock device comprises a solenoid that when activated, inhibits movement of a first component of the equipment.

3. The electronic interlock system as defined in claim 1, wherein the interlock device comprises a solenoid that when activated, inhibits rotation of a gear or cam.

4. The electronic interlock system as defined in claim 1, wherein the interlock device comprises a solenoid that when activated, isolates at least a portion of the equipment from a power source.

5. The electronic interlock system as defined in claim 1, wherein the equipment comprises a vehicle.

6. The electronic interlock system as defined in claim 1, wherein the interlock device is configured to selectively inhibit movement of a first control.

7. The electronic interlock system as defined in claim 1, the set of actions further comprising:

determine if there is sufficient sensor data to meet a first certainty threshold with respect to the impairment categorization; and

acquiring additional sensor data as a response to a determination that there is insufficient sensor data to meet the first certainty threshold with respect to the impairment categorization.

8. The electronic interlock system as defined in claim 1, wherein the determination of characteristics of the human vocal expression further comprises:

determine a pitch of the human vocal expression;

determine a volume of the human vocal expression;

determine how rapidly the first user is speaking in the human vocal expression; and

determine a magnitude and/or a power spectrum of the human vocal expression.

9. The electronic interlock system as defined in claim 1, wherein the determination of characteristics of the first user face further comprises identifying slow eye movements, drifting eye movements, deficits in saccadic movement, partial eye closures, and/or drooping eyelids.

10. The electronic interlock system as defined in claim 1, the set of actions further comprising:

use the impairment categorization for the first user to determine whether a notification is to be generated and transmitted to one or more destinations, the notification including the impairment categorization, at least a first of the one or more images, and at least a portion of the text.

11. A computer-implemented method comprising:

under control of a hardware computing device configured with specific computer-executable instructions;

receiving, over a network via a network interface from a remote device, a digitized human vocal expression of a first user and one or more digital images;

processing the received digitized human vocal expression to;

generate a power spectrum of the received digitized human vocal expression;

use the generated power spectrum to identify quiet periods in the digitized human vocal expression;

detect speech characteristics including length of pauses between words using the identified quiet periods in the digitized human vocal expression;

processing the received one or more images to detect characteristics of the first user face, including detecting if the first user has;

a sagging lip, facial droop, slow eye movements, drifting eye movements, deficits in saccadic movement, partial eye closures, drooping eyelids, and/or a dilated pupil;

using the detected speech characteristics of the first user, including at least the detected length of pauses between words, and the characteristics of the first user face to determine an impairment categorization for the first user; and

using the impairment categorization for the first user to determine whether an item of equipment is to be inhibited from use by the first user;

at least partly in response to determining that the equipment is to be inhibited from use by the first user, controlling an interlock device so as to inhibit use of the equipment by the first user.

12. The method as defined in claim 11, wherein the interlock device comprises a solenoid that when activated, inhibits movement of a first component of the equipment.

13. The method as defined in claim 11, wherein the interlock device comprises a solenoid that when activated, isolates at least a portion of the equipment from a power source.

14. The method as defined in claim 11, wherein the interlock device is configured to selectively inhibit movement of a first control.

15. The method as defined in claim 11, the method further comprising:

determining if there is sufficient sensor data to meet a first certainty threshold with respect to the impairment categorization; and

acquiring additional sensor data as a response to determining there is insufficient sensor data to meet the first certainty threshold with respect to the impairment categorization.

16. The method as defined in claim 11, wherein determining characteristics of the human vocal expression further comprises:

determining a pitch of the human vocal expression;

determining a volume of the human vocal expression;

determining how rapidly the first user is speaking in the human vocal expression; and

determining a magnitude and/or power spectrum of the human vocal expression.

17. The method as defined in claim 11, the method further comprising using the impairment categorization for the first user to determine whether a notification is to be transmitted to one or more destinations, the notification including the impairment categorization, at least a first of the one or more images, and at least a portion of the text.

18. A non-transitory computer-readable storage medium storing computer-executable instructions that when executed by a processor perform operations comprising:

receiving a digitized human vocal expression of a first user and one or more digital images of the first user;

processing the received digitized human vocal expression to;

generate a power spectrum of the received digitized human vocal expression;

use the generated power spectrum to identify quiet periods in the digitized human vocal expression;

detect speech characteristics of the digitized human vocal expression including;

length of pauses between words using the identified quiet periods in the digitized human vocal expression;

processing the received one or more images to detect characteristics of the first user face relating to user impairment;

using the detected speech characteristics, including at least the detected length of pauses between words, and the detected characteristics of the first user face to determine an impairment categorization for the first user; and

using the impairment categorization for the first user, determining whether the first user is to be inhibited from performing a first action; and

at least partly in response to determining that the first user is to be inhibited from performing a first action, generating a signal configured to cause the first user to be inhibited from performing the first action.

19. The non-transitory computer-readable storage medium of claim 18, wherein the signal configured to cause the first user to be inhibited from taking a first action controls a solenoid to inhibit movement of a first component.

20. The non-transitory computer-readable storage medium of claim 18, wherein the signal configured to cause the first user to be inhibited from taking a first action controls a solenoid to isolate at least a first item of equipment from a power source.

21. The non-transitory computer-readable storage medium of claim 18, wherein the signal configured to cause the first user to be inhibited from taking a first action controls an interlock device configured to selectively inhibit activation of a first control.

22. The non-transitory computer-readable storage medium of claim 18, the operations further comprising:

determining if there is sufficient sensor data to meet a first certainty threshold with respect to the impairment categorization; and

acquiring additional sensor data as a response to determining there is insufficient sensor data to meet the first certainty threshold with respect to the impairment categorization.

23. The non-transitory computer-readable storage medium of claim 18, wherein determining characteristics of the human vocal expression further comprises:

determining a pitch of the human vocal expression;

determining a volume of the human vocal expression;

determining how rapidly the first user is speaking in the human vocal expression; and

determining a magnitude and/or power spectrum of the human vocal expression.

24. The non-transitory computer-readable storage medium of claim 18, the operations further comprising using the impairment categorization for the first user to determine whether a notification is to be transmitted to one or more destinations, the notification including the impairment categorization and a video recording of the first user or a link thereto.

25. A system, comprising:

a network interface;

at least one computing device;

computer readable memory including instructions operable to be executed by the at least one computing device to perform a set of actions, configuring the at least one computing device to;

receive, via the network interface, a digitized human vocal expression of a first user and one or more digital images;

process the received digitized human vocal expression to;

generate a power spectrum of the received digitized human vocal expression;

use the generated power spectrum to identify quiet periods in the digitized human vocal expression;

detect speech characteristics of the digitized human vocal expression including at least;

length of pauses between words using the identified quiet periods in the digitized human vocal expression;

process the received one or more images to detect characteristics of the first user face relating to user impairment;

use the detected speech characteristics and the detected characteristics of the first user face to determine an impairment categorization for the first user; and

use the impairment categorization for the first user to determine whether the first user is to be inhibited from performing a first action; and

at least partly in response to a determination that the first user is to be inhibited from performing a first action, generate a signal configured to cause the first user to be inhibited from performing the first action.

26. The system as defined claim 25, wherein the signal configured to cause the first user to be inhibited from taking a first action is configured to control a solenoid to inhibit movement of a first component and/or isolate at least a first item of equipment from a power source.

27. The system as defined claim 25, the operations further comprising:

determine if there is sufficient sensor data to meet a first certainty threshold with respect to the impairment categorization; and

acquire additional sensor data as a response to a determination that here is insufficient sensor data to meet the first certainty threshold with respect to the impairment categorization.

28. The system as defined claim 25, wherein the determination of characteristics of the human vocal expression further comprises:

determine a pitch of the human vocal expression;

determine a volume of the human vocal expression;

determine how rapidly the first user is speaking in the human vocal expression; and

determine a magnitude and/or a power spectrum of the human vocal expression.

29. The system as defined claim 25, the operations further comprising:

use the impairment categorization for the first user to determine whether a notification is to be transmitted to one or more destinations, the notification including the impairment categorization and a video recording of the first user or a link thereto.

30. The system as defined claim 25, the operations further:

monitor the first user for impairment while the first user is operating an item of equipment.