NEURO-COGNITIVE DRIVER STATE PROCESSING

• US 20130325202A1
• Filed: 06/01/2012
• Published: 12/05/2013
• Est. Priority Date: 06/01/2012
• Status: Abandoned Application

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: HOWARD, Michael D., BHATTACHARYYA, Rajan, DAILY, Michael J.
• Application Number:
  US13/486,224
• Publication Number:
  US 20130325202A1
• Time in Patent Office: Days
• US Class Current: 701/1
• CPC Class Codes: B60W 2050/0071 Controller overrides driver automatically
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 40/09 Driving style or behaviour

Abstract

A driver state module for interfacing with a vehicle, with a surroundings vicinity of the vehicle and with a driver of the vehicle, the driver state module comprising: (i) a frame memory for storing representations of behaviors with related context; (ii) an evaluation system for ranking the frames based on goals and rewards; (iii) a working memory comprising a foreground sub-memory and a background sub-memory, the working memory for holding and sorting frames into foreground and background frames, and (iv) a recognition processor for identifying salient features relevant to a frame in the foreground memory ranked highest by the evaluation system.

Claims

1. A driver state module for interfacing with a vehicle, with a surrounding vicinity of the vehicle and with a driver of the vehicle, the driver state module comprising:

(i) a frame memory for storing representations of behaviors with related context;

(ii) an evaluation system for ranking the frames based on goals and rewards;

(iii) a working memory comprising

a foreground sub-memory, a background sub-memory, and a control for sorting frames into the foreground sub-memory or the background sub-memory; and

(iv) a recognition processor for identifying salient features relevant to a frame in the foreground sub-memory or the background sub-memory ranked highest by the evaluation system.

2. The driver state module of claim 1, configured for modeling focus of attention and awareness of the driver and for predicting imminent actions of the driver.

3. The driver state module of claim 1 wherein said interfacing with the vehicle, the surrounding vicinity of the vehicle and the driver of the vehicle is via sensors.

4. A vehicle comprising the driver state module of claim 1.

5. A driver assistance system for assisting a driver of a vehicle within a surrounding vicinity of the vehicle, the driver assistance system comprising:

(i) the driver state module of claim 1;

(ii) a vehicle state module for describing state of the vehicle in the surrounding vicinity;

(iii) a mismatch detection module for comparing the driver state module and the vehicle state module and for assessing whether there is a mismatch between the driver state module and the vehicle state module;

(iv) a driver associate interface module for determining a required action if the vehicle state module detects a mismatch, and

(v) a sensor pre-processing module for fusing data from a plurality of sensors on the vehicle and for outputting fused data in formats appropriate to each module.

6. The driver assistance system of claim 5, wherein the driver state module comprises:

(i) a frame memory for storing representations of behaviors with related context;

(ii) an evaluation system for ranking the frames based on goals and rewards;

(iii) a working memory comprising

a foreground sub-memory, a background sub-memory, and a control for sorting frames into the foreground sub-memory or the background sub-memory; and

(iv) a recognition processor for identifying salient features relevant to a frame in the foreground sub-memory or background sub-memory ranked highest by the evaluation system.

7. The driver assistance system of claim 6 configured for an application including at least one of:

(i) controlling the vehicle for short periods of time whilst the driver is distracted;

(ii) semi-autonomously controlling of the vehicle;

(iii) receiving feedback from driver behavior for self-learning by experience;

(iv) learning driving characteristics of a particular driver to optimize response to the particular driver,

(v) modeling focus of attention and awareness of the driver and

(vi) predicting imminent actions of the driver.

8. The driver assistance system of claim 5 wherein said plurality of sensors comprises at least one vehicle sensor for sensing vehicle related parameters.

9. The driver assistance system of claim 8 wherein the at least one vehicle sensor is selected from the group consisting of speed gauges, engine temperature gauges, fuel gauges, rev counters, sensors that note whether windscreen wipers are deployed, sensors that note whether lights are deployed, sensors that note whether hazard systems are deployed, and sensors that note the position of the steering wheel.

10. The driver assistance system of claim 5 wherein said plurality of sensors comprises at least one driver sensor for sensing driver related parameters.

11. The driver assistance system of claim 10 wherein said at least one driver sensor is selected from the group the group consisting of sensors for sensing the driver'"'"'s awareness, cameras providing feedback of driver'"'"'s alertness from nodding, cameras providing feedback of driver'"'"'s alertness from eye closing, eye trackers for tracking driver'"'"'s attention from direction of gaze, steering wheel mounted pressure sensors, galvanic skin response sensors for monitoring perspiration and electroencephalography sensors.

12. The driver assistance system of claim 5 wherein said plurality of sensors comprises at least one vicinity sensor for sensing variables relating to a surrounding vicinity of the vehicle.

13. The driver assistance system of claim 12 wherein said at least one vicinity sensor is selected from the group from the group consisting of forward looking cameras, lane following sensors, distance sensors deployed in all directions to determine distance of nearby objects, radar, sonar, IR sensors, general position sensors, GPS, ambient temperature sensors and ambient light sensors.

14. The driver assistance system of claim 5 configured for use in at least one application selected from the group consisting of semi-autonomous control, accident prevention, alerting, education, driver simulation and vehicle design optimization.

15. A vehicle comprising the driver assistance system of claim 5.

16. A computer software product that includes a medium readable by a processor, the medium having stored thereon:

(i) a first set of instructions for storing representations of behaviors with related context as frames in a memory;

(ii) a second set of instructions for ranking the frames based on goals and rewards;

(iii) a third set of instructions for holding and sorting the frames into foreground frames and background frames, and

(iv) a fourth set of instructions for identifying salient features relevant to a foreground frame having a highest ranking.

17. A computer software product that includes a medium readable by a processor, the medium having stored thereon a set of instructions for assisting a driver of a vehicle within a surrounding vicinity of the vehicle, comprising:

(a) a first set of instructions which, when loaded into main memory and executed by a processor models focus of attention and awareness of the driver for predicting imminent actions of the driver;

(b) a second set of instructions which, when loaded into main memory and executed by a processor describe the state of the vehicle in the surrounding vicinity;

(c) a third set of instructions which, when loaded into main memory and executed by a processor describe comparing results obtained from the first and second sets of instructions for assessing whether there is a mismatch requiring further action;

(d) a fourth set of instructions which, when loaded into main memory and executed by a processor determine the required action if running the third set of instructions detects a mismatch, and

(e) a fifth set of instructions which, when loaded into main memory and executed by a processor, fuse data from a plurality of sensors on the vehicle and outputs the fused data in formats appropriate to each of first, second, third and fourth sets of instructions.

18. A method for interfacing with a vehicle, with a surrounding vicinity of the vehicle and with a driver of the vehicle, comprising:

(i) storing representations of driver behaviors with related context as frames in a frame memory;

(ii) ranking the frames based on goals and rewards;

(iii) holding and sorting frames into foreground and background frames within a working memory, and

(iv) identifying salient features relevant to the frame with a highest ranking.

19. A method for processing sensor inputs from a plurality of sensors on a vehicle relating to a driver, the vehicle and a surrounding vicinity, comprising:

(b) fusing data from the plurality of sensors and outputting the fused data in appropriate formats;

(i) modeling focus of attention and awareness of the driver for predicting imminent actions of the driver,

(ii) describing a state of the vehicle in the surrounding vicinity;

(iii) comparing results obtained from the predicted imminent actions and the state of the vehicle to determine mismatches;

(iv) assessing whether there is a mismatch requiring further action, and

(v) determining the required action if a mismatch is detected.