Method and system for detecting operator alertness

• US 7,692,551 B2
• Filed: 02/07/2007
• Issued: 04/06/2010
• Est. Priority Date: 09/12/2006
• Status: Active Grant

• Current Assignee: Deere & Company
• Original Assignee: Deere & Company
• Inventors: Bonefas, Zachary Thomas, Sanchez, Julian
• Primary Examiner(s): Bugg; George A
• Application Number:
  US11/672,202
• Publication Number:
  US 20080068187A1
• Time in Patent Office: 1,154 Days
• Field of Search: 340/575, 340/576, 340/439, 340/573.1, 340/540, 382/2, 382/117, 382/203, 382/291, 434/236, 351/206, 351/209, 351/246
• US Class Current: 340/575
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

A method and system for detecting operator alertness of an operator of a vehicle or machine comprises an image processor and an analyzer. The image processor determines whether at least one eye of an operator is visible and determines whether the operator is generally facing forward with respect to the vehicle. The analyzer applies at least one of position analysis and motion analysis to a three dimensional representation of an operator to determine alertness of the operator if at least one eye of operator is not visible or If the operator is generally facing forward.

Claims

1. A method for detecting operator alertness of an operator of a vehicle or machine, the method comprising steps of:

determining whether at least one eye of the operator is visible;

determining whether the operator is generally facing forward with respect to the vehicle;

and responsive to determining that at least one eye of the operator is not visible or that the operator is not facing forward, applying at least one of position analysis and motion analysis to a three dimensional representation of the operator to determine alertness of the operator, wherein the applying further comprises;

collecting reference position data associated with the operator, or a portion thereof, when the operator is in an alert state;

determining observed position data of one or more points of a three dimensional representation of the operator during a time interval, wherein the three dimensional representation of the operator is established via stereo vision processing; and

sending an alert signal to alert the operator if a detected angular shift of one or more points of the representation exceed a displacement threshold.

2. The method according to claim 1 further comprising:

applying an eye monitoring algorithm to detect operator alertness, in lieu of the step of applying at least one of position analysis and motion analysis, if at least one eye of the operator is visible and if the operator is facing forward.

3. The method according to claim 1 further comprising:

gathering a pair of raw stereo scene images of the operator, or the portion thereof, from spatially offset perspectives;

combining the raw stereo scene images to produce a disparity map image that represents a disparity between the raw stereo scene images; and

creating a three dimensional representation of the operator, or the portion thereof, that excludes non-operator related background image data based on the disparity map and the raw stereo scene images.

4. The method according to claim 1 wherein the detected angular shift represents an inclination forwards, backwards, or sideways of one or more points of the representation.

5. The method according to claim 1 wherein the applying further comprises:

collecting reference motion data associated with the operator representation when the operator is in an alert state;

determining observed motion data of one or more points of a three dimensional representation of the operator during a time interval, wherein a change in position versus time is determined for the one or more points of the three dimensional representation of the operator; and

sending an alert signal to alert the operator if a detected angular shift of one or more points of the representation exceed a motion threshold.

6. The method according to claim 5 further comprising establishing the three dimensional representation of the operator via stereo vision processing.

7. The method according to claim 6 further comprising:

gathering a pair of raw stereo scene images of the operator, or the portion thereof, from spatially offset perspectives;

combining the raw stereo scene images to produce a disparity map image that represents a disparity between the raw stereo scene images; and

creating a three dimensional representation of the operator, or the portion thereof, that excludes non-operator related background image data based on the disparity map and the raw stereo scene images.

8. The method according to claim 5 wherein the detected angular shift represents an inclination forwards, backwards, or sideways of one or more points of the representation during an elapsed time period.

9. A system for detecting operator alertness of an operator of a vehicle or machine, the system comprising:

an image processor for determining whether at least one eye of the operator is visible and for determining whether the operator is generally facing forward with respect to the vehicle; and

an analyzer for applying, responsive to determining that at least one eye of the operator is not visible or that the operator is not facing forward, at least one of position analysis and motion analysis to a three dimensional representation of the operator to determine alertness of the operator if at least one eye of operator is not visible or if the operator is generally facing forward, wherein the applying further comprises;

an image collection system for collecting reference position data associated with the operator, or portion thereof, when the operator is in an alert state;

an object position estimator for determining observed position data of one or more points of a three dimensional representation of the operator during a time interval;

a three dimensional image former for establishing the three dimensional representation of the operator via stereo vision processing; and

the analyzer for sending an alert signal to alert the operator if a detected angular shift of one or more points of the representation exceed a displacement threshold.

10. The system according to claim 9 further comprising:

an evaluator for applying an eye monitoring algorithm, in lieu of the analyzer applying at least one of position analysis and motion analysis, to detect operator alertness if at least one eye of the operator is visible and if the operator is generally facing forward.

11. The system according to claim 9 further comprising:

a plurality of imaging units for gathering a pair of raw stereo scene images of the operator, or the portion thereof, from spatially offset perspectives;

an image disparity module for combining the raw stereo scene images to produce a disparity map image that represents a disparity between the raw stereo scene images; and

a three dimensional image former for creating a three dimensional representation of the operator representation that excludes non-operator related background image data based on the disparity map and the raw stereo scene images.

12. The system according to claim 9 wherein the detected angular shift represents an inclination forwards, backwards, or sideways of one or more points of the representation.

13. The system according to claim 9 further comprising an object detection module for identifying the operator, or the portion thereof, in a cab or cockpit by analyzing one or more images in three dimensional color space.

14. The system according to claim 9 wherein the applying further comprises:

an image collection system for collecting reference motion data associated with the operator, or a portion thereof, when the operator is in an alert state;

an object motion estimator for determining observed motion data of one or more points of a three dimensional representation of the operator during a time interval, wherein a change in position versus time is determined for the one or more points of the three dimensional representation of the operator;

the analyzer for sending an alert signal to alert the operator if a detected angular shift of one or more points of the representation exceed a motion threshold.

15. The system according to claim 14 further comprising a three dimensional image former for establishing the three dimensional representation of the operator via stereo vision processing.

16. The system according to claim 14 further comprising:

a plurality of imaging units for gathering a pair of raw stereo scene images of the operator, or the portion thereof, from spatially offset perspectives;

an image disparity module for combining the raw stereo scene images to produce a disparity map image that represents a disparity between the raw stereo scene images; and

a three dimensional image former for creating a three dimensional representation of the operator, or the portion thereof, that excludes non-operator related background image data based on the disparity map and the raw stereo scene images.

17. The system according to claim 14 wherein the detected angular shift represents an inclination forwards, backwards, or sideways of one or more points of the representation during a time period.

18. The system according to claim 14 further comprising an object detection module for identifying the operator, or the portion thereof, in a cab or cockpit by analyzing one or more images in three dimensional color space.