METHOD AND ARRANGEMENT FOR INTERPRETING A SUBJECTS HEAD AND EYE ACTIVITY

• US 20050073136A1
• Filed: 10/15/2003
• Published: 04/07/2005
• Est. Priority Date: 10/15/2002
• Status: Active Grant

• Current Assignee: Volvo Technology Corporation
• Original Assignee: Volvo Technology Corporation
• Inventors: Victor, Trent, Larsson, Petter
• Granted Patent: US 7,460,940 B2
• Time in Patent Office: Days
• US Class Current: 280/735
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/1103 Detecting eye twinkling
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/6821 Eye
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

Method for analyzing ocular and/or head orientation characteristics of a subject. A detection and quantification of the position of a driver'"'"'s head and/or eye movements are made relative to the environment. Tests of the data are made, and from the data, locations of experienced areas/objects-of-subject-interest are deduced. When a driver of a vehicle is the subject, these areas/objects-of-driver-interest may be inside or outside the vehicle, and may be constituted by (1) “things” such as audio controls, speedometers and other gauges, and (2) areas or positions such as “road ahead” and lane-change clearance space in adjacent lanes. In order to “standardize” the tracking data with respect to the vehicle of interest, the quantification of the position of the driver'"'"'s head is normalized to the reference-base position thereby enabling deducement of location(s) where the driver has shown an interest based on sensed information regarding either, or both of (1) driver ocular orientation or (2) driver head orientation.

Claims

1. A method for analyzing ocular and/or head orientation characteristics of a driver of a vehicle, said method comprising:

detecting and quantifying the position of a driver'"'"'s head relative to the space within a passenger compartment of a vehicle;

providing a reference-base position of a driver'"'"'s head thereby enabling the cross-reference of locations of areas/objects-of-driver-interest relative thereto; and

normalizing said quantification of the position of the driver'"'"'s head to said reference-base position thereby enabling deducement of location(s) of driver interest based on sensed information regarding at least one of (1) driver ocular orientation and (2) driver head orientation.

2. The method as recited in claim 1, further comprising:

preferentially utilizing sensed information regarding driver ocular orientation as basis for said deducement of location(s) of driver interest; and

switching to utilization of sensed information regarding driver head orientation as basis for said deducement of location(s) of driver interest when the quality of said sensed information regarding driver ocular orientation degrades beyond a prescribed threshold gaze confidence level.

3. The method as recited in claim 2, wherein said threshold gaze confidence level is exceeded when the driver'"'"'s eyes are occluded.

4. The method as recited in claim 2, wherein said threshold gaze confidence level is exceeded when the driver'"'"'s head orientation departs away from an eyes-forward orientation beyond an allowed degree of deviation.

5. The method as recited in claim 1, further comprising:

utilizing a mathematic transformation for accomplishing said normalization of said quantification of the position of the driver'"'"'s head to said reference-base position.

6. The method as recited in claim 5, further comprising:

performing said mathematic transformation using a vehicle-based computer on a substantially real time basis.

7. The method as recited in claim 1, further comprising:

prescribing probable positions of areas/objects-of-driver-interest relative to said reference-base position.

8. The method as recited in claim 1, further comprising:

defining probable positions of areas/objects-of-driver-interest relative to said reference-base position based on sensed driver ocular characteristics.

9. The method as recited in claim 8, further comprising:

establishing said definitions of probable positions of areas/objects-of-driver-interest relative to said reference-base position based on the sensed driver ocular characteristic of gaze frequency.

10. The method as recited in claim 9, further comprising:

quantifying said establishment of said gaze frequency based on collected gaze density characteristics.

13. The method as recited in claim 1, wherein said sensed information regarding driver ocular orientation is exclusively constituted by a measure of gaze angularity.

14. The method as recited in claim 1, wherein said sensed information regarding driver ocular orientation includes a measure of gaze angularity.

15. The method as recited in claim 13, wherein said measure of gaze angularity is derived from a sensed eyeball-orientation-based gaze-direction vector.

16. The method as recited in claim 1, further comprising:

defining probable positions of areas/objects-of-driver-interest relative to said reference-base position based on sensed head orientation.

17. The method as recited in claim 16, further comprising:

establishing said definitions of probable positions of areas/objects-of-driver-interest relative to said reference-base position based on sensed head orientation from which a face-forward direction is deduced.

18. The method as recited in claim 17, further comprising:

collecting a plurality of data points, each referencing a particular sensed head orientation and hence a face-forward direction, and based upon said data points, establishing density mappings indicative of frequency at which a driver looks in a certain direction.

19. The method as recited in claim 16, further comprising:

identifying an object/area-of-driver-interest by correlating said mapping to prescribed/defined probable locations of areas/objects-of-driver-interest relative to said reference-base position.

20. The method as recited in claim 19, further comprising:

tailoring prescribed functionalities performed by the vehicle based on said correlation.

21. The method as recited in claim 16, wherein said sensed information regarding head orientation is exclusively constituted by a measure of face-forward direction angularity.

22. The method as recited in claim 16, wherein said sensed information regarding head orientation includes a measure of face-forward direction angularity.

23. The method as recited in claim 21, wherein said measure of gaze angularity is derived from a sensed head-orientation-based gaze-direction vector.

11. The method as recited in claim, further comprising:

identifying an object-of-driver-interest based on driver ocular characteristics by mapping said sensed driver ocular characteristics to said prescribed or defined probable locations of areas/objects-of-driver-interest relative to said reference-base position.

12. The method as recited in claim 11, further comprising:

tailoring prescribed functionalities performed by the vehicle based on said mapped driver ocular characteristics.

24-66. -66. (cancelled)

67. A method for developing a bench-mark for comparison in assessing driver activity and/or driver condition, said method comprising:

collecting a stream of gaze-direction data based on a sensed characteristic of a driver;

ascertaining a region representative of typical eyes-forward driving based on a high-density pattern assessed from said collected gaze-direction data; and

utilizing said collected gaze-direction data, in comparison to said ascertained representative region of typical eyes-forward driving, to identify and assess the severity of at least one of the following driver impairment characteristics based on said comparison;

(1) cognitive driver distraction, (2) visual driver distraction, and (3) high driver work load.

68. The method as recited in claim 67, further comprising:

utilizing measures of at least one of (1) driver ocular orientation and (2) driver head orientation to constitute said gaze-direction data.

69. The method as recited in claim 67, further comprising:

calculating a percentage road center (PRC) driver characteristic based on the relative amount of time, during a prescribed time period, that eyes-forward driving is maintained by the driver.

70. The method as recited in claim 69, further comprising:

assessing a condition of high driver work load based on a PRC calculation indicative of high driver time-sharing activity.

71. The method as recited in claim 69, further comprising:

assessing a condition of cognitive driver distraction based on a PRC calculation sufficiently high to be indicative of driver preoccupation characterized by staring ahead in the eyes forward orientation with insufficient looking away.

72. The method as recited in claim 69, further comprising:

assessing a condition of visual driver distraction based on a PRC calculation sufficiently low to be indicative of driver attention diversion characterized by looking away from the eyes forward orientation for too great a percentage of the prescribed time period.

73. The method as recited in claim 67, further comprising:

adjusting an area of said ascertained region representative of typical eyes-forward driving dependent upon a sensed driving condition.

74. The method as recited in claim 73, wherein said sensed driving condition is at least one of (1) vehicle speed and (2) driving environment.

75. The method as recited in claim 67, further comprising:

logging identified incidents of cognitive distraction, visual distraction and high driver workload.

76. The method as recited in claim 75, further comprising:

storing said logged identified incidents for future analysis.

77. The method as recited in claim 67, further comprising:

transmitting said logged identified incidents to a processor for further analysis.

78. The method as recited in claim 77, further comprising:

performing said transmittal and said analysis on a real-time basis.

79. The method as recited in claim 69, further comprising:

providing driver feedback when a PRC-based severity quantification of driver impairment exceeds a prescribed severity threshold level.

80. The method as recited in claim 69, further comprising:

tailoring prescribed functionalities performed by the vehicle when a PRC-based severity quantification of driver impairment exceeds a prescribed severity threshold level.

81. A method for developing a bench-mark for comparison in assessing driver activity and/or driver condition, said method comprising:

collecting a stream of gaze-direction data based on a sensed characteristic of a driver, and based on density patterns developed therefrom, defining at least two areas of driver interest;

analyzing said stream of collected gaze-direction data utilizing a primary moving time-window of prescribed period; and

detecting multiple driver glances between said at least two target areas within said primary moving time-window indicative of an occurrence of high driver time-sharing activity.

82. The method as recited in claim 81, further comprising:

identifying periods of high driver workload based on a frequency of threshold-exceeding occurrences of driver time-sharing activity.

83. The method as recited in claim 81, further comprising:

refreshing said primary moving time-window upon the detection of cessation of an occurrence of driver time-sharing activity.

84. A method for automated analysis of eye movement data, said method comprising:

processing data descriptive of eye movements observed in a subject using a computer-based processor by applying classification rules to the data and thereby identifying at least visual fixations experienced by the subject;

analyzing gaze-direction information associated with the identified fixations thereby developing data representative of directions in which the subject visually fixated during the period of data collection;

segregating the developed data, based at least partially on fixation gaze-direction, into delimited data sets, each delimited data set representing an area/object-of-subject-interest existing during the period of data collection, and at least one of said delimited data sets representing a region of typical eyes-forward driving based on a high-density pattern assessed from said gaze-direction information; and

calculating a percentage road center (PRC) driver characteristic from the developed data representing a relative quantification of driver maintained, eyes-forward driving during a prescribed period of time.

85. The method as recited in claim 84, further comprising:

utilizing said developed data, in comparison to said delimited data set defining the representative region of typical eyes-forward driving, to identify and assess the severity of at least one of the following driver impairment characteristics based on said comparison;

(1) cognitive driver distraction, (2) visual driver distraction, and (3) high driver work load.

86. The method as recited in claim 84, further comprising:

identifying glances by applying at least one glance-defining rule to the data, each of said identified glance encompassing at least one identified fixation.

87. The method as recited in claim 86, further comprising:

basing said at least one glance-defining rule on at least one characteristic selected from the group including;

glance duration, glance frequency, total glance time, and total task time.

88. The method as recited in claim 84, further comprising:

segregating said identified glances into delimited glance sets based at least partially on a gaze-direction during the respective glance, each of said segregated glance sets representing an area/object-of-subject-interest existing during the period of data collection.

89. The method as recited in claim 88, further comprising:

assessing a relative density of one glance set in comparison to at least one other glance set, and based thereupon, identifying the represented area/object-of-subject-interest of the compared glance set.

90. The method as recited in claim 88, further comprising:

assessing a relative density of at least one glance set among a plurality of glance sets, and based upon a mapping of said assessed relative density to known relative densities associated with settings of the type in which the eye movement data was collected, identifying the represented area/object-of-subject-interest of the compared glance set.

91. The method as recited in claim 88, further comprising:

assessing relative densities of at least two glance sets developed from data descriptive of eye movements observed in a known setting; and

identifying the represented area/object-of-subject-interest of each of the two compared glance sets and ascertaining locations of said represented areas/objects-of-subject-interest in the known setting thereby establishing a special reference frame for the known setting.

92. The method as recited in claim 91, wherein said subject is a driver of a vehicle and based on a density of at least one of the glance data sets, an eyes-forward, normal driver eye orientation is deduced.

93. The method as recited in claim 84, wherein said applied classification rules comprise at least criteria defining fixations and transitions.

94. The method as recited in claim 84, wherein said applied classification rules further comprise criteria defining saccades.

95. The method as recited in claim 84, wherein said subject is a driver of a vehicle and the method further comprises utilizing a plurality of analysis protocols dependent upon prevailing noise characteristics associated with the data set being processed.

96. The method as recited in claim 95, further comprising:

applying a first data filter of predetermined stringency to an input stream of data comprising said data descriptive of eye movements observed in a driver of a vehicle utilizing said computer-based processor and therefrom outputting a first filtered data stream corresponding to said input stream of data; and

assessing quality of said outputted first filtered data stream by applying a first approval rule thereto, and data of said outputted first filtered data stream passing said first approval rule being outputted and constituting an approved first stream of data.

97. The method as recited in claim 96, further comprising:

applying a second data filter of greater stringency than said first data filter to the input stream of data utilizing said computer-based processor and therefrom outputting a second filtered data stream corresponding to said a first filtered data stream via common derivation from the input stream of data; and

assessing quality of said outputted second filtered data stream by applying a second approval rule thereto, and data of said outputted second filtered data stream passing said second approval rule being outputted and constituting a approved second stream of data.

98. The method as recited in claim 97, further comprising:

composing a collective approved stream of data constituted by an entirety of said approved first stream of data, and said collective approved stream of data being further constituted by portions of said approved second stream of data corresponding to unapproved portions of said outputted first filtered data stream.

99. The method as recited in claim 97, wherein said first and second approval rules are the same.

100. The method as recited in claim 97, wherein said first and second approval rules are based on the same criteria.

101. The method as recited in claim 95, further comprising:

selecting at least two analysis protocols to constitute said plurality from a group consisting of;

(1) a velocity based, dual threshold protocol that is best suited, relative to the other members of the group, to low-noise-content eye behavior data;

(2) a distance based, dispersion spacing protocol that is best suited, relative to the other members of the group, to moderate-noise-content eye and eyelid behavior data; and

(3) an ocular characteristic based, rule oriented protocol that is best suited, relative to the other members of the group, to high-noise-content eye behavior data.

102. The method as recited in claim 101, wherein said selection of protocols for any given data set is biased toward one of said three protocols in dependence upon a detected noise level in the data set.

103. The method as recited in claim 101, wherein said rule oriented protocol considers one or more of the following standards in a discrimination between fixations and saccades:

(1) fixation duration must exceed 150 ms;

(2) saccade duration must not exceed 200 ms; and

saccades begin and end in two different locations.

104. The method as recited in claim 95, further comprising:

assessing quality of said data descriptive of eye movement based on relative utilization of respective analysis protocols among said plurality of analysis protocols.

105. The method as recited in claim 93, further comprising:

assessing quality of said data descriptive of eye movement considering time-based, relative utilization of respective analysis protocols among said plurality of analysis protocols over a prescribed time period.

106. The method as recited in claim 84, further comprising:

analyzing a stream of collected driver eye-gaze data utilizing a stream-traversing primary time-window of prescribed period and detecting an artifact that clouds the trueness of a portion of the data stream; and

resorting to a secondary moving time-window simultaneously traversing said data stream and generating highly filtered data from said collected data when said artifact is encountered.

107. The method as recited in claim 84, further comprising:

analyzing a stream of collected driver eye-gaze data utilizing a stream-traversing primary time-window of prescribed period; and

detecting characteristics within said primary time-window indicative of data quality-degradation beyond a prescribed quality threshold level during data stream traversal.

108. The method as recited in claim 107, further comprising:

resorting to a secondary moving time-window simultaneously traversing said data stream and generating highly filtered data from said collected data when said data quality-degradation exceeds the prescribed quality threshold level.

109. The method as recited in claim 108, further comprising:

returning to said primary moving time-window when said data quality-degradation is detected to have subsided within the prescribed quality threshold level.