Method and device for monitoring at least one vehicle occupant, and method for operating at least one assistance device

• US 10,140,532 B2
• Filed: 12/09/2011
• Issued: 11/27/2018
• Est. Priority Date: 04/12/2011
• Status: Active Grant

• Current Assignee: Daimler AG (Mercedes-Benz Group AG)
• Original Assignee: Daimler AG (Mercedes-Benz Group AG)
• Inventors: Weidl, Galia, Herrtwich, Ralf, Schrauf, Michael
• Primary Examiner(s): Williams, Jeffery A
• Application Number:
  US14/111,440
• Publication Number:
  US 20140104405A1
• Time in Patent Office: 2,545 Days
• Field of Search: 348 77
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness

Abstract

A method and device for monitoring at least one vehicle passenger in a vehicle involves capturing images of the vehicle passenger using an image capturing unit and analyzing the captured images using an image processing unit. At least one vital sign of the vehicle passenger is determined by the image analysis of the captured images, which can be used to operate at least one assistance device of a vehicle.

Claims

1. A method for monitoring at least one vehicle passenger in a vehicle, the method comprising:

capturing images of the vehicle passenger using at least one camera directed toward a first bodily region;

analyzing the captured images of the first bodily region using at least one processor;

determining a first vital sign and a second vital sign different from the first vital sign of the vehicle passenger, the determining of each of the first and second vital signs being based on the image analysis of the captured images;

determining, based on the determined first vital sign and the second vital sign, one or more diagnostic values for;

(i) tachycardia or bradycardia, (ii) heart rate variability, and (iii) tendency related to viral infection; and

recommending a suitable countermeasure based at least on the determined one or more diagnostic values,

wherein the determined first vital sign is a pulse rate and the second vital sign is a breathing rate, and

wherein the first vital sign and the second vital sign are both determined from the same captured images of the first bodily region so that it is unnecessary to move or rotate the at least one camera toward a second bodily region to determine the second vital sign.

2. The method according to claim 1, wherein one or more of the first and second vital signs is determined by

detecting an optical absorption value or an intensity value of reflected light of a skin region of the vehicle passenger, or

calculating a change of the optical absorption value or of the intensity value.

3. The method according to claim 1, wherein during image processing for analyzing the captured images, at least one region suited to image analysis is looked for in each of the captured images and only the at least one region suited to image analysis is analyzed.

4. The method according to claim 3, wherein the at least one region suited to image analysis contains a face of the vehicle passenger.

5. The method according to claim 3, wherein the captured images of the vehicle passenger are color, wherein the region of the images suited to image analysis is split into individual color channels and at least one of the individual color channels is analyzed to determine one or more of the first and second vital signs.

6. The method according to claim 5, wherein a spectral power density of the at least one of the individual color channels of the region suited to image analysis is determined using a Fast-Fourier Transform.

10. The method according to claim 1, further comprising the step of distinguishing whether the first vital sign indicates calm, stress, or elation based on the one or more diagnostic values related to the heart rate variability.

11. The method according to claim 1, wherein the suitable countermeasure includes one or more of the following:

(i) taking a refreshment break, (ii) moving regularly, (iii) taking sufficient fluid, and (iv) taking required medication.

12. The method according to claim 1, further comprising activating at least one function of one or more devices of the vehicle based on the determined first vital sign and the second vital sign.

13. The method according to claim 1, further comprising providing the recommendation of the suitable countermeasure to the at least one vehicle passenger of the vehicle via one or more devices of the vehicle and the at least one vehicle passenger acting on the recommendation based on the recommendation provided by the one or more devices of the vehicle.

7. A device for monitoring at least one vehicle passenger in a vehicle, the device comprising:

at least one camera directed toward a first bodily region and configured to capture images of the vehicle passenger; and

at least one processor executing stored instructions to;

analyze the captured images of the first bodily region,

determine a first vital sign and a second vital sign different from the first vital sign of the vehicle passenger, the determination of each of the first and second vital signs being based on image analysis of the captured images,

determine, based on the determined first vital sign and the second vital sign, one or more diagnostic values for;

(i) tachycardia or bradycardia, (ii) heart rate variability, and (iii) tendency related to viral infection, and

recommend a suitable countermeasure based at least on the determined one or more diagnostic values,

wherein the determined first vital sign is a pulse rate and the second vital sign is a breathing rate, and

wherein the first vital sign and the second vital sign are both determined from the same captured images of the first bodily region so that it is unnecessary to move or rotate the at least one camera toward a second bodily region to determine the second vital sign.

8. A method, comprising:

capturing images of a vehicle passenger in a vehicle using at least one camera directed toward a first bodily region;

analyzing the captured images of the first bodily region using at least one processor;

determining a first vital sign and a second vital sign different from the first vital sign of the vehicle passenger, the determining of each of the first and second vital signs being based on the image analysis of the captured images;

determining, based on the determined first vital sign and the second vital sign, one or more diagnostic values for;

(i) tachycardia or bradycardia, (ii) heart rate variability, and (iii) tendency related to viral infection; and

activating at least one function of an assistance device if a determined value of one or more of the first and second vital signs deviates from a predetermined value or value region and recommending a suitable countermeasure based at least on the determined one or more diagnostic values,

wherein the determined first vital sign is a pulse rate and the second vital sign is a breathing rate, and

wherein the first vital sign and the second vital sign are both determined from the same captured images of the first bodily region so that it is unnecessary to move or rotate the at least one camera toward a second bodily region to determine the second vital sign.

9. The method according to claim 8, wherein an intervention in a steering device, braking device, or in a drive train of the vehicle, an optical, audible, or haptic warning device, or at least one comfort function or an emergency function is activated as a function of the assistance device.