Warning a vehicle operator of unsafe operation behavior based on a 3D captured image stream

• US 7,792,328 B2
• Filed: 01/12/2007
• Issued: 09/07/2010
• Est. Priority Date: 01/12/2007
• Status: Expired due to Fees

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Goldman, Steven D., Sessa, Anthony J., Arnold, Kenneth C., Paolini, Michael A., Albertson, Jacob C.
• Primary Examiner(s): KIM, CHONG R
• Application Number:
  US11/622,684
• Publication Number:
  US 20080169914A1
• Time in Patent Office: 1,334 Days
• Field of Search: None
• US Class Current: 382/104
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

A computer-implemented method, system, and program product includes a behavior processing system for capturing a three-dimensional movement of an operator of a vehicle, wherein the three-dimensional movement is determined by using at least one image capture device aimed at the operator, and identifying at least one behavior of the operator from the captured three-dimensional movement. An operator monitoring system for a vehicle determines whether the predicted behavior and the current operational state of the vehicle indicate unsafe operation of the vehicle and the operator monitoring system notifies the operator when unsafe operator of the vehicle is indicated by the operator'"'"'s behavior for the current operational state of the vehicle.

Claims

1. A computer-implemented method for automated detection of unsafe operation of a vehicle and automated response to detected unsafe operator of a vehicle, comprising:

capturing a three-dimensional movement of an operator of a vehicle, wherein the three-dimensional movement is determined using at least one image capture device aimed at the operator;

identifying a three-dimensional object properties stream using the captured movement;

identifying a particular defined behavior of the operator represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions;

identifying a current operational state of the vehicle;

identifying whether the particular defined behavior and the current operational state of the vehicle match at least one unsafe operation definition from among a plurality of unsafe operation definitions;

in response to matching the at least one unsafe operation definition, notifying the operator of the type of unsafe operation currently performed based on the at least one unsafe operation definition;

prompting the operator via an audio output interface of the vehicle to perform a particular step from among a plurality of operation training steps;

detecting the particular defined behavior of the operator and the current operational state of the vehicle;

determining whether the particular defined behavior matches an expected behavior in the particular step and whether the current operational state of the vehicle matches an expected operational state in the particular step;

responsive to determining the particular defined behavior matches the expected behavior in the particular step and the current operational state of the vehicle matches the expected operational state in the particular step, generating an output signal for output via the audio output interface indicating compliance and recording a record of the compliance;

responsive to determining at least one the particular defined behavior does not match the expected behavior in the particular step and the current operational state of the vehicle does not match the expected operational state in the particular step, determining whether a behavior translation database specifies triggering a training warning signal and at least one instruction for adjusting behavior for the particular defined behavior and the current operational state; and

responsive to the behavior translation database specifying triggering the training warning signal for the particular defined behavior and the current operational state, outputting the training warning signal and the at least one instruction for adjusting behavior to ensure the safety of the operator during training.

2. The method according to claim 1, wherein capturing a three-dimensional movement of the operator of a vehicle further comprises capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the operator.

3. The method according to claim 1, wherein identifying a particular defined behavior of the operator representing the three-dimensional object properties stream further comprises calculating a percentage probability that the captured three-dimensional movement represents a particular behavior defined in the particular behavior definition.

4. The method according to claim 1, wherein identifying a particular defined behavior of the operator representing the three-dimensional object properties stream further comprises identifying at least one of body movement of the operator comprising a fine granularity movement of at least one of a facial expression, an eye movement, a muscle contraction, and change in skin surface characteristics of the operator within the three-dimensional object properties stream matching at least one of the plurality of behavior definitions.

5. The method according to claim 1, wherein identifying whether the particular defined behavior and the current operational state of the vehicle match at least one unsafe operation definition from among a plurality of unsafe operation definitions, further comprises identifying whether a 3D position of the head and the eyes of the operator during the current operational state of the vehicle matches the unsafe 3D position of the head and eyes for the current operational state defined in the at least one unsafe operation definition.

6. The method according to claim 1, wherein identifying whether the particular defined behavior and the current operational state of the vehicle match at least one unsafe operation definition from among a plurality of unsafe operation definitions, further comprises identifying whether the detected behavior of the operator talking on a portable telephone device during the current operational state of the vehicle matches the unsafe portable telephone use behavior for the current operational state defined in the at least one unsafe operation definition.

7. The method according to claim 1, wherein notifying the operator of the type of unsafe operation currently performed based on the at least one unsafe operation definition further comprises:

generating at least one control signal from among a tactile output control signal, an audio output control signal and a graphical output control signal, wherein the at least one control signal controls output that notifies the operator of the type of unsafe operation; and

controlling output of the at least one control signal to a controller for controlling the output that notifies the operator the type of unsafe operation.

8. The method according to claim 1, further comprising:

recording the particular defined behavior in a storage system separate from the vehicle via a network, responsive to the current operational state specifying at least one from among an airbag deployment, an antilock brake system actuation, and detecting damage to at least one part of the vehicle.

9. A system for automated detection of unsafe operation of a vehicle and automated response to detected unsafe operator of a vehicle, comprising:

a behavior processing system communicatively connected to a network, further comprising;

means for capturing a three-dimensional movement of an operator of a vehicle, wherein the three-dimensional movement is determined using at least one image capture device aimed at the operator;

means for identifying a three-dimensional object properties stream using the captured movement;

means for identifying a particular defined behavior of the operator represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions;

an operator monitoring system communicatively connected to the behavior processing system via the network, further comprising;

means for identifying a current operational state of the vehicle;

means for identifying whether the particular defined behavior and the current operational state of the vehicle match at least one unsafe operation definition from among a plurality of unsafe operation definitions;

means, in response to matching the at least one unsafe operation definition, for notifying the operator of the type of unsafe operation currently performed based on the at least one unsafe operation definition;

an operator training system for prompting the operator via an audio output interface of the vehicle to perform a particular step from among a plurality of operation training steps;

the operator training system for detecting the particular defined behavior of the operator and the current operational state of the vehicle;

the operator training system for determining whether the particular defined behavior matches an expected behavior in the particular step and whether the current operational state of the vehicle matches an expected operational state in the particular step;

the operator training system, responsive to determining the particular defined behavior matches the expected behavior in the particular step and the current operational state of the vehicle matches the expected operational state in the particular step, for generating an output signal for output via the audio output interface indicating compliance and recording a record of the compliance;

the operator training system, responsive to determining at least one the particular defined behavior does not match the expected behavior in the particular step and the current operational state of the vehicle does not match the expected operational state in the particular step, for determining whether a behavior translation database specifies triggering a training warning signal for the particular defined behavior and the current operational state; and

the operator training system, responsive to the behavior translation database specifying triggering the training warning signal for the particular defined behavior and the current operational state, for outputting the training warning signal to ensure the safety of the operator during training.

10. The system according to claim 9, wherein the means for capturing a three-dimensional movement of the operator of a vehicle further comprises means for capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the operator.

11. The system according to claim 9, wherein the means for identifying a particular defined behavior of the operator representing the three-dimensional object properties stream further comprises means for identifying at least one of body movement of the operator comprising a fine granularity movement of at least one of a facial expression, an eye movement, a muscle contraction, and change in skin surface characteristics of the operator within the three-dimensional object properties stream matching at least one of the plurality of behavior definitions.

12. The system according to claim 9, wherein the means for identifying whether the particular defined behavior and the current operational state of the vehicle match at least one unsafe operation definition from among a plurality of unsafe operation definitions, further comprises means for identifying whether a 3D position of the head and the eyes of the operator during the current operational state of the vehicle matches the unsafe 3D position of the head and eyes for the current operational state defined in the at least one unsafe operation definition.

13. The system according to claim 9, wherein the means for identifying whether the particular defined behavior and the current operational state of the vehicle match at least one unsafe operation definition from among a plurality of unsafe operation definitions, further comprises means for identifying whether the detected behavior of the operator talking on a portable telephone device during the current operational state of the vehicle matches the unsafe portable telephone use behavior for the current operational state defined in the at least one unsafe operation definition.

14. The system according to claim 9, wherein the means for notifying the operator of the type of unsafe operation currently performed based on the at least one unsafe operation definition further comprises:

means for generating at least one control signal from among an audio output control signal and a graphical output control signal, wherein the at least one control signal controls output that notifies the operator of the type of unsafe operation; and

means for controlling output of the at least one control signal to a controller for controlling the output that notifies the operator the type of unsafe operation.

15. The system according to claim 9, the operator monitoring system further comprising means for recording the particular defined behavior in a storage system separate from the vehicle via a network, responsive to the current operational state specifying at least one from among an airbag deployment, an antilock brake system actuation, and detecting damage to at least one part of the vehicle.

16. A program product comprising a non-transitory storage-type computer-readable medium including a computer-readable program for automated detection of unsafe operation of a vehicle and automated response to detected unsafe operator of a vehicle, wherein the computer-readable program when executed on a computer causes the computer to:

capture a three-dimensional movement of an operator of a vehicle, wherein the three-dimensional movement is determined using at least one image capture device aimed at the operator;

identify a three-dimensional object properties stream using the captured movement;

identify a particular defined behavior of the operator represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions;

identify a current operational state of the vehicle;

identify whether the particular defined behavior and the current operational state of the vehicle match at least one unsafe operation definition from among a plurality of unsafe operation definitions;

in response to matching the at least one unsafe operation definition, notify the operator of the type of unsafe operation currently performed based on the at least one unsafe operation definition;

prompt the operator via an audio output interface of the vehicle to perform a particular step from among a plurality of operation training steps;

detect the particular defined behavior of the operator and the current operational state of the vehicle;

determine whether the particular defined behavior matches an expected behavior in the particular step and whether the current operational state of the vehicle matches an expected operational state in the particular step;

responsive to determining the particular defined behavior matches the expected behavior in the particular step and the current operational state of the vehicle matches the expected operational state in the particular step, generate an output signal for output via the audio output interface indicating compliance and recording a record of the compliance;

responsive to determining at least one the particular defined behavior does not match the expected behavior in the particular step and the current operational state of the vehicle does not match the expected operational state in the particular step, determine whether a behavior translation database specifies triggering a training warning signal and at least one instruction for adjusting behavior for the particular defined behavior and the current operational state; and

responsive to the behavior translation database specifying triggering the training warning signal for the particular defined behavior and the current operational state, output the training warning signal and the at least one instruction for adjusting behavior to ensure the safety of the operator during training.

17. The program product of claim 16, wherein the computer-readable program when executed on a computer further causes the computer to capture the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the operator.