Vehicle occupant monitoring using infrared imaging

• US 9,988,055 B1
• Filed: 08/26/2016
• Issued: 06/05/2018
• Est. Priority Date: 09/02/2015
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: O'Flaherty, Seth, Mahaffey, Sean, Kumar, Ayush, Schroeder, Jacob
• Primary Examiner(s): Wang, Jack K
• Application Number:
  US15/248,073
• Time in Patent Office: 648 Days
• Field of Search: 340576
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/043 Identity of occupants
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 2556/55 using telemetry
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: G08B 13/19647 Systems specially adapted for intrusion detection in or around a vehicle
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: H04W 4/90 Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]

Abstract

Methods and systems for monitoring vehicle occupants using infrared and other sensors are disclosed. The systems may use infrared sensor data to identify each vehicle occupant using biometric signatures (such as heartbeat or facial recognition). Vehicle occupant characteristics may be determined that include skeletal characteristics of the occupants. Further monitoring of the occupants using sensor data may occur until an abnormal situation is detected. Abnormal situations may include medical emergencies, driver impairment, security threats, or similar situations requiring correcting action. The system may then determine and implement an appropriate response to the abnormal situation. Such responses may include generating alerts, adjusting vehicle environmental controls, taking control of operation of the vehicle (such as an autonomous or semi-autonomous vehicle), or initiating wireless communication with an outside party, such as an emergency service.

Claims

1. A computer-implemented method for monitoring one or more vehicle occupants of a vehicle, comprising:

receiving, from one or more sensors disposed within the vehicle, sensor data regarding the one or more vehicle occupants;

determining one or more vehicle occupant characteristics for at least one of the one or more vehicle occupants based upon the received sensor data, wherein the at least one of the one or more vehicle occupants is distinct from a vehicle operator, wherein the one or more vehicle occupant characteristics include one or more skeletal characteristics of the at least one of the one or more vehicle occupants;

determining whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the at least one of the one or more vehicle occupants, wherein the abnormal situation relates to one or more of the following types of abnormal situations;

a medical emergency, a health risk, an accident risk, an impairment of a vehicle occupant, or a security threat;

determining one or more responses to the abnormal situation based upon the one or more determined vehicle occupant characteristics when an abnormal situation is determined to exist, wherein the one or more responses are based upon the determined type of the abnormal situation; and

causing the one or more responses to the abnormal situation to be implemented.

2. The computer-implemented method of claim 1, wherein the one or more skeletal characteristics indicate the position of a plurality of segments of the vehicle occupant'"'"'s body, wherein the plurality of segments include the vehicle occupant'"'"'s head and at least a portion of a limb of the vehicle occupant.

3. The computer-implemented method of claim 1, further comprising:

identifying the at least one of the one or more vehicle occupants based upon the one or more determined vehicle occupant characteristics, wherein identifying the at least one of the one or more vehicle occupants includes comparing the one or more determined vehicle occupant characteristics with data regarding characteristics stored in a user profile.

4. The computer-implemented method of claim 3, wherein determining whether the abnormal situation exists includes determining whether the one or more determined vehicle occupant characteristics are beyond a baseline range for the vehicle occupant based upon the data regarding the characteristics stored in the user profile of the vehicle occupant.

5. The computer-implemented method of claim 1, wherein the one or more responses include one or more of the following:

controlling vehicle operation by an on-board computer system, adjusting an environmental condition within the vehicle, communicating a message to an emergency response service, or terminating vehicle operation.

6. The computer-implemented method of claim 1, wherein the one or more responses include communicating sensor data to one or more computing devices associated with emergency response personnel.

7. The computer-implemented method of claim 1, wherein the one or more sensors include one or more infrared sensors disposed within the vehicle.

8. A computer system for monitoring one or more vehicle occupants of a vehicle, comprising:

one or more processors;

one or more sensors disposed within the vehicle and communicatively connected to the one or more processors; and

a program memory coupled to the one or more processors and storing executable instructions that when executed by the one or more processors cause the computer system to;

receive sensor data regarding the one or more vehicle occupants from the one or more sensors;

determine one or more vehicle occupant characteristics for at least one of the one or more vehicle occupants based upon the received sensor data, wherein the at least one of the one or more vehicle occupants is distinct from a vehicle operator, wherein the one or more vehicle occupant characteristics include one or more skeletal characteristics of the at least one of the one or more vehicle occupants;

determine whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the at least one of the one or more vehicle occupants, wherein the abnormal situation relates to one or more of the following types of abnormal situations;

a medical emergency, a health risk, an accident risk, an impairment of a vehicle occupant, or a security threat;

determine one or more responses to the abnormal situation based upon the one or more determined vehicle occupant characteristics when an abnormal situation is determined to exist, wherein the one or more responses are based upon the determined type of the abnormal situation; and

cause the one or more responses to the abnormal situation to be implemented.

9. The computer system of claim 8, wherein the one or more skeletal characteristics indicates the position of a plurality of segments of the vehicle occupant'"'"'s body, wherein the plurality of segments include the vehicle occupant'"'"'s head and at least a portion of a limb of the vehicle occupant.

10. The computer system of claim 8, wherein the program memory further stores executable instructions that cause the computer system to:

identify the at least one of the one or more vehicle occupants based upon the one or more determined vehicle occupant characteristics, wherein identifying the at least one of the one or more vehicle occupants includes comparing the one or more determined vehicle occupant characteristics with data regarding characteristics stored in a user profile.

11. The computer system of claim 10, wherein the executable instructions that cause the computer system to determine whether the abnormal situation exists further cause the computer system to determine whether the one or more determined vehicle occupant characteristics are beyond a baseline range for the vehicle occupant based upon the data regarding the characteristics stored in the user profile of the vehicle occupant.

12. The computer system of claim 8, wherein the one or more responses include one or more of the following:

controlling vehicle operation by an on-board computer system, adjusting an environmental condition within the vehicle, communicating a message to an emergency response service, or terminating vehicle operation.

13. The computer system of claim 8, wherein the one or more sensors include one or more infrared sensors disposed within the vehicle.

14. A tangible, non-transitory computer-readable medium storing instructions for monitoring one or more vehicle occupants of a vehicle that, when executed by at least one processor of a computer system, cause the computer system to:

receive sensor data regarding the one or more vehicle occupants from one or more sensors disposed within the vehicle;

determine one or more vehicle occupant characteristics for at least one of the one or more vehicle occupants based upon the received sensor data, wherein the at least one of the one or more vehicle occupants is distinct from a vehicle operator, wherein the one or more vehicle occupant characteristics include one or more skeletal characteristics of the at least one of the one or more vehicle occupants;

determine whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the at least one of the one or more vehicle occupants, wherein the abnormal situation relates to one or more of the following types of abnormal situations;

a medical emergency, a health risk, an accident risk, an impairment of a vehicle occupant, or a security threat;

determine one or more responses to the abnormal situation based upon the one or more determined vehicle occupant characteristics when an abnormal situation is determined to exist, wherein the one or more responses are based upon the determined type of the abnormal situation; and

cause the one or more responses to the abnormal situation to be implemented.

15. The tangible, non-transitory computer-readable medium of claim 14, wherein the one or more skeletal characteristics of the at least one of the one or more vehicle occupants indicate the position of a plurality of segments of the vehicle occupant'"'"'s body, wherein the plurality of segments include the vehicle occupant'"'"'s head and at least a portion of a limb of the vehicle occupant.

16. The tangible, non-transitory computer-readable medium of claim 14, further storing instructions that cause the computer system to:

identify the at least one of the one or more vehicle occupants based upon the one or more determined vehicle occupant characteristics, wherein identifying the at least one of the one or more vehicle occupants includes comparing the one or more determined vehicle occupant characteristics with data regarding characteristics stored in a user profile.

17. The tangible, non-transitory computer-readable medium of claim 16, wherein the instructions that cause the computer system to determine whether the abnormal situation exists further cause the computer system to determine whether the one or more determined vehicle occupant characteristics for a vehicle operator are beyond a baseline range for the vehicle occupant based upon the data regarding the characteristics stored in the user profile of the vehicle operator.

18. The tangible, non-transitory computer-readable medium of claim 14, wherein the one or more responses include one or more of the following:

controlling vehicle operation by an on-board computer system, adjusting an environmental condition within the vehicle, communicating a message to an emergency response service, or terminating vehicle operation.