Head-mounted display head pose and activity estimation

• US 9,767,373 B2
• Filed: 09/05/2014
• Issued: 09/19/2017
• Est. Priority Date: 09/05/2014
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Yang, Hsin-hsiang, Prakah-Asante, Kwaku O.
• Primary Examiner(s): MERCEDES, DISMERY E
• Application Number:
  US14/478,440
• Publication Number:
  US 20160070966A1
• Time in Patent Office: 1,110 Days
• Field of Search: None
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: B60R 21/01542 detecting passenger motion
• CPC Class Codes: B60R 21/01552 detecting position of specific human body parts, e.g. face, eyes or hands
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0827 due to sleepiness
• CPC Class Codes: B60W 2050/0055 High-pass filters
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2422/00 Indexing codes relating to the special location or mounting of sensors
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system may receive head pose indications determined according to movement data from a motion sensor of an optical head-mounted display worn by a vehicle driver, determine, according to the head pose indications, driver activity characteristics indicative of a history of movement of the head of the driver, and send the driver activity characteristics to a driver-aware vehicle system configured to adjust driver notification based on the driver activity characteristics. The system may also receive raw movement data from a motion sensor of an optical head-mounted display worn on a head of a vehicle driver, compute head velocity data based on the raw movement data, compute head displacement data based on the head velocity data, and update head pose indications indicative of head positioning upon determining that the head displacement data exceeds a predetermined threshold displacement.

Claims

1. A system comprising:

a processor configured to

store head pose indications,

receive raw movement data from a motion sensor of an optical head-mounted display worn on a head of a vehicle driver,

compute head velocity data based on the raw movement data,

compute head displacement data based on the head velocity data,

update the stored head pose indications indicative of head positioning upon determining that the head displacement data exceeds a predetermined threshold displacement,

determine, according to the head pose indications over time, driver activity characteristics indicative of a history of frequency and magnitude of movement of the head of the driver, and

send the driver activity characteristics to a driver-aware vehicle system configured to adjust driver workload notification settings based on driver-action previously observed with respect to the driver activity characteristics.

2. The system of claim 1, wherein the head pose indications are further determined according to information received from an in-vehicle camera monitoring the head of the driver.

3. The system of claim 1, wherein the driver-aware vehicle system includes a customized content display system configured to adjust display of content provided in the optical head-mounted display according to the driver activity characteristics.

4. The system of claim 1, wherein the driver-aware vehicle system includes a workload estimator configured to determine driver workload based on the driver activity characteristics.

5. The system of claim 1, wherein the driver-aware vehicle system includes a driver drowsiness module configured to estimate driver drowsiness based on the driver activity characteristics.

6. A system comprising:

a processor configured to

store head pose indications,

receive raw movement data from a motion sensor of an optical head-mounted display worn on a head of a vehicle driver,

compute head velocity data based on the raw movement data,

compute head displacement data based on the head velocity data, and

update the stored head pose indications indicative of head positioning upon determining that the head displacement data exceeds a predetermined threshold displacement.

7. The system of claim 6, wherein the processor is further configured to determine, according to the head pose indications, driver activity characteristics indicative of a history of movement of the head of the driver.

8. The system of claim 6, wherein the processor is further configured to perform a high-pass filter on the raw movement data to adjust for data bias before computing the head velocity data.

9. The system of claim 6, wherein the processor is further configured to perform a high-pass filter on the head displacement data to adjust for data bias before updating the head pose indications.

10. The system of claim 6, wherein the raw movement data includes indications of head position, and the head displacement data includes indications of head displacement along an x-axis of the head, a y-axis of the head, and a z-axis of the head.

11. The system of claim 10, wherein the predetermined threshold displacement includes at least one of a threshold displacement along the x-axis of the head, a threshold displacement along the y-axis of the head, and a threshold displacement along the z-axis of the head.

12. A system comprising:

a processor configured to

store head pose indications,

receive raw movement data from a motion sensor of an optical head-mounted display worn on a head of a vehicle driver,

compute head velocity data based on the raw movement data,

compute head displacement data based on the head velocity data,

update the stored head pose indications indicative of head positioning upon determining that the head displacement data exceeds a predetermined threshold displacement,

determine a history of head pose indications over time according to the movement data,

determine likely driver activity characteristics indicative of frequency and magnitude of driver head movement based on an observed history of the head pose indications, and

adjust driver workload according to the driver activity characteristics.

13. The system of claim 12, wherein the head pose indications are further determined according to information received from an in-vehicle camera monitoring the head of the driver.

14. The system of claim 12, wherein the movement data includes indications of head position, and the head pose indications includes indications of head positioning along an x-axis of the head, a y-axis of the head, and a z-axis of the head.

15. The system of claim 12, wherein the motion sensor includes at least one of an accelerometer sensor and a gyroscopic sensor.

16. The system of claim 12, wherein the processor is configured to receive the movement data from the motion sensor of the optical head-mounted display over a wireless connection between the optical head-mounted display and the processor.