Systems and methods for detecting surprising events in vehicles

• US 10,427,655 B2
• Filed: 10/21/2016
• Issued: 10/01/2019
• Est. Priority Date: 10/23/2015
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Nix, Axel
• Primary Examiner(s): Kong, Sze-Hon
• Application Number:
  US15/331,685
• Publication Number:
  US 20170113664A1
• Time in Patent Office: 1,075 Days
• CPC Class Codes: B60T 7/12 for automatic initiation; for initiation not subject to will of driver or passenger limiting speed of vehicles other than rail vehicles B60K31/00
• CPC Class Codes: B60T 8/171 Detecting parameters used in the regulation; Measuring values used in the regulation
• CPC Class Codes: B60T 8/17551 determining control parameters related to vehicle stability used in the regulation, e.g. by calculations involving measured or detected parameters
• CPC Class Codes: B60T 8/17558 specially adapted for collision avoidance or collision mitigation road vehicle drive control systems for collision avoidance otherwise than by controlling a particular sub-unit B60W30/09
• CPC Class Codes: B60T 8/241 Lateral vehicle inclination
• CPC Class Codes: B60T 8/32 responsive to a speed condition, e.g. acceleration or deceleration using electrical circuitry or regulation means B60T8/17 ; B60T8/28 takes precedence; electric devices on electrically propelled vehicles indicating the wheel slip B60L3/10; measuring linear or angular speed per se G01P3/00
• CPC Class Codes: B60W 2050/0018 Method for the design of a control system
• CPC Class Codes: B60W 2050/041 Built in Test Equipment [BITE]
• CPC Class Codes: B60W 2050/046 involving external transmission of data to or from the vehicle, e.g. via telemetry, satellite, Global Positioning System [GPS]
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4026 Cycles
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 2556/65 Data transmitted between vehicles
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 60/0016 : of the vehicle or its occupants
• CPC Class Codes: B60W 60/0051 : from occupants to vehicle
• CPC Class Codes: G06F 11/3072 : where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 : Event detection
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G07C 5/008 : communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0841 : Registering performance data recording measured values G01D; information storage G11B
• CPC Class Codes: G07C 5/0866 : the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G07C 5/0891 : Video recorder in combination with video camera
• CPC Class Codes: G08B 13/19669 : Event triggers storage or change of storage policy
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 : Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A driver assistance system is presented, including an event detector which comprises a set of rules defining a surprising event based on signals reflecting a vehicle operator input signals from an object detection sensor. An event data file generator is configured to generate an event data file according to rules comprised in the event detector, the event data file comprising a video signal received from a camera, a signal from at least one dynamic vehicle sensor, and target object information received from the object detection sensor. The event data file generator is further configured to initiate data file generation responsive to a surprising event being detected by the event generator, and wherein the contents of the data file are specified by the rules comprised in the event detector. In this way, surprising events may be collected and analyzed off-board in order to generate updates for the driver assistance system.

Claims

1. A surprising event detection system for a vehicle having a driver assistance system that controls lateral and/or longitudinal movement of the vehicle, the surprising event detection system comprising:

an interface communicatively connected to a camera, at least one dynamic vehicle sensor, and an object detection sensor;

a processor; and

one or more storage devices that store a set of rules defining a surprising event based on a combination of at least two input signals and that further store instructions executable by the processor to;

generate, with an event data file generator responsive to a surprising event being detected, an event data file comprising;

a video signal received from the camera;

a signal from the at least one dynamic vehicle sensor; and

object information received from the object detection sensor,

periodically export event data to an analytic server, and

receive an update patch from an update server and apply the update patch to the surprising event detection system, wherein the update patch comprises updated rules adjusting a spectrum of surprising events to be stored and adjusting a handling of a detection of each surprising event of the spectrum of surprising events, the adjusting of the handling of the detection of each surprising event including adjusting a weight of data received from a selected sensor responding to an indication of a change in reliability of the selected sensor.

2. The surprising event detection system as in claim 1, wherein the combination of at least two input signals comprises at least one signal reflecting a vehicle operator input and at least one signal from the object detection sensor, and wherein the driver assistance system includes an automatic emergency braking mechanism.

3. The surprising event detection system as in claim 1, wherein the combination of at least two input signals comprises at least one signal received over an extra-vehicle communication system from a second vehicle.

4. The surprising event detection system as in claim 1, wherein the combination of at least two input signals is transmitted on a vehicle data bus and is selected from the group consisting of a vehicle speed signal, a vehicle yaw rate signal, a steering wheel angle signal, a vehicle brake pedal position signal, a vehicle accelerator pedal position signal, a preceding vehicle distance signal, a preceding vehicle relative velocity signal, a turn indicator signal, a steering torque request signal, a vehicle acceleration request signal, and an airbag deployment signal.

5. The surprising event detection system as in claim 1, wherein the combination of at least two input signals comprises at least one signal from the object detection sensor and at least one signal from a second object detection sensor.

6. The surprising event detection system as in claim 1, further comprising a video compressor, wherein the video signal in the event data file has been compressed by the video compressor.

7. The surprising event detection system as in claim 1, wherein the combination of at least two input signals comprises:

at least one signal selected from the group consisting of a steering wheel angle signal, a vehicle brake pedal position signal, and a vehicle accelerator pedal position signal; and

at least one signal selected from the group consisting of a steering torque request, a steering angle request, and an acceleration request.

8. The surprising event detection system as in claim 1, wherein the set of rules defining the surprising event is associated with rules defining the event data file, and wherein the rules defining the event data file comprise:

a length of time to be recorded before detection of the surprising event,

a length of time to be recorded after detection of the surprising event,

a frame rate of video to be recorded, and/or

an identification of signals to be recorded.

9. The surprising event detection system as in claim 1,

further comprising a rolling event data buffer which receives signals from a vehicle data bus,

wherein the event data file comprises data present in the rolling event data buffer at a time of detecting the surprising event and data received by the surprising event detection system after the time of detecting the surprising event.

10. The surprising event detection system of claim 1, wherein the instructions are further executable to upload the event data file to the analytic server based on the set of rules, wherein the set of rules comprises an indication of a priority for the detected surprising event and/or a selected data transfer network for uploading the event data file to the analytic server.

11. The surprising event detection system of claim 1, wherein the object detection sensor is a radar sensor, a lidar sensor, or a machine vision camera.

12. The surprising event detection system of claim 1, wherein the event data file is asymmetrically encrypted with a public key of a designated recipient.

13. A method for detecting surprising events in a vehicle having a driver assistance system that controls lateral and/or longitudinal movement of the vehicle, the method comprising:

receiving signal data from one or more onboard vehicle sensors and one or more object identification sensors;

receiving video data from one or more video cameras;

storing the signal data and the video data in an event data buffer;

identifying a surprising event from the received signal data based on surprise event rules stored at a storage device;

deriving an event data file for the surprising event based on the surprise event rules;

uploading the event data file to a server;

receiving an update patch from the server, comprising updated surprise event rules; and

applying the update patch to adjust a packaging of event data files, to change the surprise event rules by adjusting a spectrum of events detected, and to adjust a handling of event data files according to associated assigned priorities by adjusting a weight of data received from a selected sensor responsive to an indication of a change in reliability of the selected sensor.

14. The method of claim 13, wherein receiving the signal data from the one or more onboard vehicle sensors and the one or more object identification sensors further comprises:

receiving data from an on-board radar sensor;

receiving data from an on-board machine vision camera;

receiving data from a dynamic vehicle sensor;

receiving data from a vehicle navigation system;

receiving data from another vehicle; and

/or

receiving data indicating an adjustment of a parameter of one or more vehicle actuators to be made responsive to signals received from the object identification sensors.

15. The method of claim 13, wherein receiving the video data from the one or more video cameras comprises:

receiving video from a forward facing camera mounted to a windshield;

receiving video from a forward facing camera mounted near a front of the vehicle;

receiving video data from a rearward facing side mirror replacement camera; and

/or

receiving video from a rearward facing camera mounted near a rear of the vehicle.

16. The method of claim 13, further comprising:

executing, on the server, a machine vision algorithm to determine the surprising event using the video data contained within the event data file; and

associating annotation information with the event data file.

17. The method of claim 13, wherein the surprise event rules are selected to cause, on average, no more than one event per week to be uploaded to the server.

18. A method for detecting surprising events in a vehicle having a driver assistance system that controls lateral and/or longitudinal movement of the vehicle, the method comprising:

receiving signal data from one or more onboard vehicle sensors and one or more object identification sensors;

receiving video data from one or more video cameras;

storing the signal data and the video data in an event data buffer;

identifying a surprising event from the received signal data based on surprise event rules stored at a storage device;

deriving an event data file for the surprising event based on the surprise event rules;

uploading the event data file to a server;

receiving an update patch from the server, comprising updated surprise event rules; and

applying the update patch to change a weight that is assigned to data received from a selected sensor responsive to an indication of a change in reliability of the selected sensor.

19. The method of claim 18, wherein the selected sensor is a first selected sensor, and wherein changing the weight that is assigned to data received from the first selected sensor comprises assigning a lower weight to data received from the first selected sensor relative to data received from a second selected sensor responsive to an indication of a reduction in the reliability of the first selected sensor.

20. The method of claim 19, wherein the reduction in the reliability of the first selected sensor is determined based on a geographical area of the vehicle or a driving condition of the vehicle.