Vehicle occupant monitoring using infrared imaging

• US 10,160,457 B1
• Filed: 03/09/2018
• Issued: 12/25/2018
• Est. Priority Date: 09/02/2015
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: O'Flaherty, Seth, Mahaffey, Sean, Kumar, Ayush, Schroeder, Jacob
• Primary Examiner(s): Wang, Jack K
• Application Number:
  US15/916,500
• Time in Patent Office: 291 Days
• Field of Search: 340576
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/043 Identity of occupants
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 2556/55 using telemetry
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: G08B 13/19647 Systems specially adapted for intrusion detection in or around a vehicle
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: H04W 4/90 Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]

Abstract

Methods and systems for monitoring vehicle occupants using infrared and other sensors are disclosed. The systems may use infrared sensor data to identify each vehicle occupant using biometric signatures (such as heartbeat or facial recognition). Vehicle occupant characteristics may be determined that include skeletal characteristics of the occupants. Further monitoring of the occupants using sensor data may occur until an abnormal situation is detected. Abnormal situations may include medical emergencies, driver impairment, security threats, or similar situations requiring correcting action. The system may then determine and implement an appropriate response to the abnormal situation. Such responses may include generating alerts, adjusting vehicle environmental controls, taking control of operation of the vehicle (such as an autonomous or semi-autonomous vehicle), or initiating wireless communication with an outside party, such as an emergency service.

Claims

1. A computer-implemented method for monitoring a vehicle occupant of a vehicle, the method comprising, via one or more processors:

receiving, from one or more sensors disposed within the vehicle, sensor data regarding the vehicle occupant, wherein the vehicle occupant is distinct from a vehicle operator;

determining one or more vehicle occupant characteristics for the vehicle occupant based upon the received sensor data, wherein the one or more vehicle occupant characteristics include one or more skeletal characteristics of the vehicle occupant;

determining whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the vehicle occupant, wherein the abnormal situation relates to one or more of the following types of abnormal situations;

a medical emergency, a health risk, an accident risk, an impairment of the vehicle occupant, or a security threat;

determining one or more responses to the abnormal situation based upon the determined type of the abnormal situation; and

causing the one or more responses to the abnormal situation to be implemented.

2. The computer-implemented method of claim 1, wherein the one or more skeletal characteristics indicate the position of a plurality of segments of the vehicle occupant'"'"'s body, wherein the plurality of segments include the vehicle occupant'"'"'s head and at least a portion of a limb of the vehicle occupant.

3. The computer-implemented method of claim 1, further comprising:

identifying the vehicle occupant by comparing the one or more determined vehicle occupant characteristics with data regarding characteristics stored in a user profile.

4. The computer-implemented method of claim 3, wherein determining whether the abnormal situation exists includes determining whether the one or more determined vehicle occupant characteristics are beyond a baseline range for the vehicle occupant based upon the data regarding the characteristics stored in the user profile of the vehicle occupant.

5. The computer-implemented method of claim 1, wherein the one or more responses include controlling vehicle operation by an on-board computer system.

6. The computer-implemented method of claim 1, wherein the one or more responses include adjusting an environmental condition within the vehicle.

7. The computer-implemented method of claim 1, wherein the one or more responses include communicating a message to an emergency response service.

8. The computer-implemented method of claim 1, wherein the one or more responses include terminating vehicle operation.

9. The computer-implemented method of claim 1, wherein the one or more responses include communicating sensor data to one or more computing devices associated with emergency response personnel.

10. The computer-implemented method of claim 1, wherein the one or more sensors include one or more infrared sensors disposed within the vehicle.

11. A computer system for monitoring a vehicle occupant of a vehicle, the computer system comprising:

one or more processors;

one or more sensors disposed within the vehicle and communicatively connected to the one or more processors; and

a program memory coupled to the one or more processors and storing executable instructions that when executed by the one or more processors cause the computer system to;

receive sensor data regarding the vehicle occupant from the one or more sensors, the vehicle occupant being distinct from a vehicle operator;

determine one or more vehicle occupant characteristics for the vehicle occupant based upon the received sensor data, wherein the one or more vehicle occupant characteristics include one or more skeletal characteristics of the vehicle occupant;

determine whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the vehicle occupant, wherein the abnormal situation relates to one or more of the following types of abnormal situations;

a medical emergency, a health risk, an accident risk, an impairment of the vehicle occupant, or a security threat;

determine one or more responses to the abnormal situation based upon the determined type of the abnormal situation; and

cause the one or more responses to the abnormal situation to be implemented.

12. The computer system of claim 11, wherein the one or more skeletal characteristics indicates the position of a plurality of segments of the vehicle occupant'"'"'s body, wherein the plurality of segments include the vehicle occupant'"'"'s head and at least a portion of a limb of the vehicle occupant.

13. The computer system of claim 11, wherein the program memory further stores executable instructions that cause the computer system to:

identify the vehicle occupant based upon the one or more determined vehicle occupant characteristics by comparing the one or more determined vehicle occupant characteristics with data regarding characteristics stored in a user profile.

14. The computer system of claim 13, wherein the executable instructions that cause the computer system to determine whether the abnormal situation exists further cause the computer system to determine whether the one or more determined vehicle occupant characteristics are beyond a baseline range for the vehicle occupant based upon the data regarding the characteristics stored in the user profile of the vehicle occupant.

15. The computer system of claim 11, wherein the one or more responses include controlling vehicle operation by an on-board computer system.

16. The computer system of claim 11, wherein the one or more responses include adjusting an environmental condition within the vehicle.

17. The computer system of claim 11, wherein the one or more responses include communicating a message to an emergency response service.

18. The computer system of claim 11, wherein the one or more responses include terminating vehicle operation.

19. The computer system of claim 11, wherein the one or more sensors include one or more infrared sensors disposed within the vehicle.

20. A tangible, non-transitory computer-readable medium storing instructions for monitoring a vehicle occupant of a vehicle that, when executed by at least one processor of a computer system, cause the computer system to:

receive sensor data regarding the vehicle occupant from one or more sensors disposed within the vehicle, wherein the vehicle occupant is distinct from a vehicle operator;

determine one or more vehicle occupant characteristics for the vehicle occupant based upon the received sensor data, wherein the one or more vehicle occupant characteristics include one or more skeletal characteristics of the vehicle occupant;

determine whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the vehicle occupant, wherein the abnormal situation relates to one or more of the following types of abnormal situations;

a medical emergency, a health risk, an accident risk, an impairment of the vehicle occupant, or a security threat;

determine one or more responses to the abnormal situation based upon the determined type of the abnormal situation; and

cause the one or more responses to the abnormal situation to be implemented.