System and method of depth sensor activation

• US 10,474,145 B2
• Filed: 11/08/2016
• Issued: 11/12/2019
• Est. Priority Date: 11/08/2016
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Guo, Feng, Qi, Yingyong, Bi, Ning, Jiang, Bolan, Chang, Chienchung
• Primary Examiner(s): Butler, Rodney A
• Application Number:
  US15/346,000
• Publication Number:
  US 20180129202A1
• Time in Patent Office: 1,099 Days
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60W 10/06 including control of combustion engines
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/223 Posture, e.g. hand, foot, or seat position, turned or inclined
• CPC Class Codes: B60W 2540/225 Direction of gaze
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2710/06 Combustion engines, Gas turbines
• CPC Class Codes: B60W 2900/00 Indexing codes relating to the purpose of, or problem solved of road vehicle drive control systems not otherwise provided for in groups B60W30/00
• CPC Class Codes: B60W 30/182 Selecting between different operative modes, e.g. comfort and performance modes
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 60/0051 from occupants to vehicle
• CPC Class Codes: B60W 60/0053 from vehicle to occupant
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness

Abstract

An apparatus includes a first sensor configured to generate first sensor data. The first sensor data is related to an occupant of a vehicle. The apparatus further includes a depth sensor and a processor. The depth sensor is configured to generate data corresponding to a volume associated with at least a portion of the occupant. The processor is configured to receive the first sensor data and to activate the depth sensor based on the first sensor data.

Claims

1. An apparatus comprising:

a first sensor configured to generate sensor data related to an occupant of a vehicle;

a depth sensor configured to generate depth data corresponding to a volume associated with at least a portion of the occupant; and

a processor configured to;

receive the sensor data;

activate the depth sensor, conditioned on identifying a state of the occupant based on the sensor data, to generate the depth data; and

initiate a remedial action conditioned on verifying the identified state.

2. The apparatus of claim 1, wherein the processor is configured to reduce exposure of the occupant to output from the depth sensor as compared to a device in which an active depth sensor is constantly activated.

3. The apparatus of claim 1, wherein the first sensor comprises an image sensor.

4. The apparatus of claim 1, wherein the first sensor includes an audio sensor.

5. The apparatus of claim 1, wherein the remedial action includes an alert responsive to a determination, based on the sensor data and the depth data, that the state corresponds to one or more defined states.

6. The apparatus of claim 5, wherein the depth sensor includes a laser device, and wherein the alert is provided via a display device or an audio device.

7. The apparatus of claim 1, wherein the depth sensor is configured to:

receive an activation signal from the processor, the activation signal configured to activate the depth sensor to project energy and generate the depth data; and

upon activation of the depth sensor, target a seat of the vehicle or a head of the occupant with the energy.

8. The apparatus of claim 5, wherein the alert is provided via haptic feedback.

9. The apparatus of claim 1, wherein the sensor data comprises image data corresponding to an image of the occupant, and wherein the processor is configured to classify the state by determining, based on the image data, a head pose of the occupant, features of a face of the occupant, or a combination thereof.

10. The apparatus of claim 1, wherein the sensor data comprises light level data, and wherein the processor is configured to activate the depth sensor in response to the light level data satisfying a threshold.

11. The apparatus of claim 1, wherein the first sensor, the depth sensor, and the processor are integrated into the vehicle.

12. A method of activating an active depth sensor, the method comprising:

receiving, at a processor, sensor data from a first sensor, the sensor data related to an occupant of a vehicle;

conditioned on detecting a compromised or hazardous state of the occupant based on the sensor data, sending an activation signal from the processor to the active depth sensor to cause the active depth sensor to generate depth data; and

receiving, at the processor, the depth data from the active depth sensor, the depth data corresponding to a volume associated with at least a portion of the occupant.

13. The method of claim 12, wherein the processor is configured to reduce exposure of the occupant to output from the active depth sensor as compared to a device in which a depth sensor is constantly activated, and further comprising determining, at the processor, a classification of the state based on the depth data and the sensor data.

14. The method of claim 12, further comprising, in response to determining that the state corresponds to one or more defined states, outputting an audio alert via a speaker, outputting a visual alert via a display, activating an automated driving system, deactivating an engine, deactivating the vehicle, or a combination thereof.

15. The method of claim 12, further comprising:

determining, at the processor, whether the state corresponds to one or more defined states based on the sensor data; and

determining, at the processor, to activate the active depth sensor in response to determining that the state corresponds to the one or more defined states for a period of time that satisfies a threshold.

16. The method of claim 15, wherein the one or more defined states are associated with distraction of the occupant, sleeping of the occupant, impairment of the occupant, or a combination thereof.

17. The method of claim 15, wherein the sensor data comprises image data corresponding to an image of the occupant, and wherein determining the state includes determining a head pose of the occupant based on the image data.

18. The method of claim 12, further comprising:

determining, based on the sensor data, whether a light level associated with the volume satisfies a threshold; and

determining to activate the active depth sensor in response to the light level satisfying the threshold.

19. A computer-readable storage device storing instructions that, when executed by a processor, cause the processor to:

receive sensor data from a first sensor, the sensor data related to an occupant of a vehicle;

activate a depth sensor, conditioned on identifying a state of the occupant based on the sensor data, to generate depth data; and

receive the depth data from the depth sensor, the depth data corresponding to a volume associated with at least a portion of the occupant.

20. The computer-readable storage device of claim 19, wherein:

the instructions are configured to reduce exposure of the occupant to output from the depth sensor as compared to an active depth sensor that is constantly activated, and

the instructions, when executed by the processor, further cause the processor to determine whether to classify the state as one or more defined states based on the depth data and the sensor data.

21. The computer-readable storage device of claim 20, wherein determining whether to classify the state as the one or more defined states includes combining two dimensional image data from the sensor data with the depth data to determine three dimensional data corresponding to the occupant.

22. The computer-readable storage device of claim 21, wherein combining the two dimensional image data with the depth data includes generating a three dimensional head pose of the occupant based on the two dimensional image data and the depth data.

23. The computer-readable storage device of claim 22, wherein the instructions, when executed by the processor, further cause the processor to classify the state as the one or more defined states in response to detecting that the three dimensional head pose indicates the one or more defined states for a duration of time that satisfies a threshold.

24. The computer-readable storage device of claim 23, wherein the instructions, when executed by the processor, further cause the processor to detect that the three dimensional head pose indicates the one or more defined states in response to the three dimensional head pose indicating that the occupant is focused away from a direction of operation of the vehicle.

25. The computer-readable storage device of claim 20, wherein the instructions, when executed by the processor, further cause the processor to:

identify passing of a period of time since activation of the depth sensor; and

deactivate the depth sensor in response to the period of time satisfying a threshold.

26. The computer-readable storage device of claim 20, wherein the instructions, when executed by the processor, further cause the processor to, in response to determining to classify the state as the one or more defined states, initiate a remedial action, the remedial action including outputting an audio alert via a speaker, outputting a visual alert via a display, activating an automated driving system, deactivating an engine, outputting a haptic alert, adjusting a setting of an audio or visual system of the vehicle, or a combination thereof.

27. The computer-readable storage device of claim 26, wherein the instructions, when executed by the processor, further cause the processor to deactivate the depth sensor in response to initiation of the remedial action.

28. The computer-readable storage device of claim 20, wherein the instructions, when executed by the processor, further cause the processor to, in response to determining to classify the state as the one or more defined states, initiate transmission of a message to a remote device.

29. An apparatus comprising:

means for generating sensor data related to an occupant of a vehicle;

means for generating depth data corresponding to a volume associated with at least a portion the occupant; and

means for activating the means for generating depth data, conditioned on identifying a state of the occupant based on the sensor data, to cause the means for generating to generate the depth data to enable determination of the detected state.

30. The apparatus of claim 29, further comprising means for outputting, based on the identified state, one or both of an audio alert and a visual alert, wherein:

the means for outputting is incorporated into the vehicle,

and

the means for activating is configured to reduce exposure of the occupant to output from the means for generating as compared to a device in which a depth sensor is constantly activated.