Saliency based awareness modeling

• US 9,747,812 B2
• Filed: 10/22/2014
• Issued: 08/29/2017
• Est. Priority Date: 10/22/2014
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Misu, Teruhisa
• Primary Examiner(s): Iwamaye, Andrew
• Assistant Examiner(s): Fletcher, Jerry-Daryl
• Application Number:
  US14/521,167
• Publication Number:
  US 20160117947A1
• Time in Patent Office: 1,042 Days
• Field of Search: 434 29, 434 62, 434 65
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 50/085 Changing the parameters of the control units, e.g. changing limit values, working points by control input
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G09B 19/16 Control of vehicles or other craft simulators G09B9/02
• CPC Class Codes: G09B 19/167 Control of land vehicles
• CPC Class Codes: G09B 9/04 for teaching control of land vehicles

Abstract

In one or more embodiments, driver awareness may be calculated, inferred, or estimated utilizing a saliency model, a predictive model, or an operating environment model. An awareness model including one or more awareness scores for one or more objects may be constructed based on the saliency model or one or more saliency parameters associated therewith. A variety of sensors or components may detect one or more object attributes, saliency, operator attributes, operator behavior, operator responses, etc. and construct one or more models accordingly. Examples of object attributes associated with saliency or saliency parameters may include visual characteristics, visual stimuli, optical flow, velocity, movement, color, color differences, contrast, contrast differences, color saturation, brightness, edge strength, luminance, a quick transient (e.g., a flashing light, an abrupt onset of a change in intensity, brightness, etc.).

Claims

1. A system for saliency based awareness modeling, comprising:

a sensor component detecting one or more objects within an operating environment and one or more object attributes for one or more of the objects, wherein one or more of the object attributes are associated with saliency of one or more of the objects;

a monitoring component detecting one or more operator attributes of an operator of a vehicle;

a modeling component constructing;

a saliency model for one or more of the objects based on one or more of the attributes associated with saliency of one or more of the objects; and

an awareness model for one or more of the objects based on the saliency model and one or more of the operator attributes; and

a scoring component assigning one or more awareness scores to one or more objects of the awareness model based on the saliency model and one or more of the operator attributes,

wherein the sensor component, the monitoring component, the modeling component, or the scoring component is implemented via a processing unit.

2. The system of claim 1, comprising an electronic control unit (ECU) receiving one or more operator responses or operator behavior associated with the operator of the vehicle.

3. The system of claim 2, wherein the modeling component constructs the awareness model based on one or more of the operator responses.

4. The system of claim 2, comprising a database component housing baseline operator response information, wherein the modeling component constructs the awareness model based on a comparison between the baseline operator response information and one or more of the operator responses.

5. The system of claim 1, comprising a notification component generating one or more notifications based on one or more awareness scores for one or more of the objects.

6. The system of claim 5, comprising a management component controlling a timing, a color, or a size of one or more of the notifications.

7. The system of claim 1, wherein the sensor component comprises an image capture device, a radar sensor, a light detection and ranging (LIDAR) sensor, a laser sensor, a video sensor, or a movement sensor.

8. The system of claim 1, wherein the monitoring component comprises an image capture sensor, a motion sensors, an eye tracking unit, an infrared sensor, an infrared illuminator, or a depth sensor.

9. The system of claim 1, wherein one or more of the object attributes comprises velocity, color, contrast, color saturation, brightness, or a detected transient for one or more of the objects.

10. The system of claim 1, wherein one or more of the operator attributes comprises eye movement, head movement, focus, facial trajectory, eye gaze trajectory, or gaze distribution.

11. A method for saliency based awareness modeling, comprising:

detecting, using a sensor component, one or more objects within an operating environment;

detecting one or more object attributes for one or more of the objects, wherein one or more of the object attributes are associated with saliency of one or more of the objects;

detecting one or more operator attributes of an operator of the vehicle;

receiving one or more operator responses provided by the operator of the vehicle;

constructing a saliency model for one or more of the objects based on one or more of the attributes associated with saliency of one or more of the objects;

constructing an awareness model for one or more of the objects based on the saliency model, one or more of the operator responses, and one or more of the operator attributes; and

assigning one or more awareness scores to one or more objects of the awareness model based on the saliency model, one or more of the operator responses, and one or more of the operator attributes,

wherein the detecting, the receiving, the constructing, or the assigning is implemented via a processing unit.

12. The method of claim 11, comprising constructing the awareness model based on a comparison between baseline operator response information and one or more of the operator responses.

13. The method of claim 11, comprising constructing the awareness model based on a comparison between baseline object attribute information and one or more of the object attributes.

14. The method of claim 11, comprising constructing the awareness model based on a comparison between baseline operator attribute information and one or more of the operator attributes.

15. The method of claim 11, comprising rendering one or more notifications based on one or more awareness scores for one or more of the objects during navigation from an origin location to a destination location.

16. The method of claim 15, comprising managing one or more aspects of one or more of the notifications.

17. A system for saliency based awareness modeling, comprising:

a sensor component detecting one or more objects within an operating environment and one or more object attributes for one or more of the objects, wherein one or more of the object attributes are associated with saliency of one or more of the objects;

a monitoring component detecting one or more operator attributes of an operator of a vehicle;

a modeling component constructing;

a saliency model for one or more of the objects based on one or more of the attributes associated with saliency of one or more of the objects; and

an awareness model for one or more of the objects based on the saliency model and one or more of the operator attributes;

a scoring component assigning one or more awareness scores to one or more objects of the awareness model based on the saliency model and one or more of the operator attributes; and

a notification component generating one or more notifications based on one or more awareness scores for one or more of the objects,

wherein the sensor component, the monitoring component, the modeling component, the scoring component, or the notification component is implemented via a processing unit.

18. The system of claim 17, wherein the sensor component is a gaze detection device tracking eye movement or gaze distribution.

19. The system of claim 17, comprising an electronic control unit (ECU) determining a number of attention demanding objects based on user interaction with one or more subunits of the ECU.

20. The system of claim 19, wherein the modeling component constructs the awareness model based on the number of attention demanding objects.