SYSTEMS AND METHODS FOR COMPLEX EVENT PROCESSING OF VEHICLE INFORMATION AND IMAGE INFORMATION RELATING TO A VEHICLE

• US 20110060496A1
• Filed: 08/10/2010
• Published: 03/10/2011
• Est. Priority Date: 08/11/2009
• Status: Active Grant

• Current Assignee: Certusview Technologies LLC (Dycom Industries Incorporated)
• Original Assignee: Certusview Technologies LLC (Dycom Industries Incorporated)
• Inventors: Nielsen, Steven, Chambers, Curtis, Farr, Jeffrey
• Granted Patent: US 8,560,164 B2
• Time in Patent Office: Days
• US Class Current: 701/33
• CPC Class Codes: G06Q 10/0631 Resource planning, allocation, distributing or scheduling for enterprises or organisations
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A vehicle-based complex event processing (CEP) engine executes on a hardware computer processor in a vehicle. The CEP engine receives a first input stream onto which vehicle information related to the vehicle is published, and applies at least one query to the first input stream. The CEP engine then assesses, based on application of the query/queries, if a vehicle-related event has occurred and, if so, the CEP engine publishes an indication of occurrence of the vehicle-related event on at least one output stream of the CEP engine. In one example, the CEP engine may receive a second input stream onto which image information is published, wherein the image information is based at least in part on data provided by at least one on-board vehicle camera.

Claims

1. A method performed by a complex event processing (CEP) engine that executes on at least one hardware computer processor in a vehicle, the method comprising:

A) receiving at least two input streams comprising;

at least one first input stream onto which vehicle information related to the vehicle is published; and

at least one second input stream onto which image information is published, wherein the image information is based at least in part on data provided by at least one on-board vehicle camera;

B) applying at least one query to the at least one first input stream and the at least one second input stream;

C) assessing, based on B), if a vehicle-related complex event has occurred; and

D) if it is determined in C) that the vehicle-related complex event has occurred, publishing an indication of occurrence of the vehicle-related complex event on at least one output stream of the CEP engine.

2. The method of claim 1, further comprising:

generating at least one perceivable alert in the vehicle based at least in part on D).

3. The method of claim 1, further comprising:

altering operation of the vehicle based at least in part on D).

4. The method of claim 1, further comprising:

transmitting the indication of the occurrence of the vehicle-related complex event from the vehicle to at least one office at which fleet management operations occur and/or to at least one supervisor responsible for the vehicle.

5. The method of claim 1, further comprising:

transmitting the indication of the occurrence of the vehicle-related complex event to at least one second CEP engine external to the vehicle.

6. The method of claim 1, wherein in A), the vehicle information comprises:

vehicle readings information regarding the vehicle provided by an Electronic Control Unit (ECU) of the vehicle.

7. The method of claim 6, wherein the vehicle readings information includes a plurality of values corresponding to a plurality of parameter identifiers defined by the On Board Diagnostic (OBD-II) standard SAE J1979.

8. The method of claim 6, wherein the vehicle readings information includes at least one of:

a vehicle identification number;

an odometer reading;

an engine temperature;

an oil temperature;

an engine water temperature;

a fuel level;

an engine RPM reading;

a vehicle speed;

a throttle depression reading;

a brake pedal application indicator;

at least one airbag indicator;

at least one seatbelt indicator;

a key position indicator;

a gear shift indicator;

a check engine indicator;

an overdrive indicator;

a parking brake indicator;

at least one door open indicator;

at least one vehicle ignition security indicator;

a windshield wiper activation indicator;

a left turn signal indicator;

a right turn signal indicator;

an emergency signal indicator;

a headlamps activation indicator;

a vehicle audio system status indicator; and

at least one timestamp indicating when vehicle readings are obtained.

9. The method of claim 1, wherein in A), the vehicle information includes non-image based environmental information and/or operational information comprising at least one of:

GPS information from at least one GPS receiver associated with the vehicle;

accelerometer information from at least one accelerometer associated with the vehicle;

humidity information from at least one humidity sensor associated with the vehicle;

temperature information from at least one temperature sensor associated with the vehicle;

light-level information from at least one light sensor associated with the vehicle;

ignition voltage information from at least one ignition voltage sensor associated with the vehicle;

cellular diagnostic information relating to cellular connectivity with the vehicle; and

wireless diagnostic information relating to WiFi connectivity with the vehicle.

10. The method of claim 1, wherein the image information includes unprocessed image information obtained from at least one output of the at least one on-board vehicle camera.

11. The method of claim 1, wherein the image information includes status information regarding operation of the at least one on-board vehicle camera.

12. The method of claim 1, wherein the image information includes processed image information derived from the data provided by the at least one on-board vehicle camera.

13. The method of claim 12, wherein the processed image information includes at least one of:

color detection information;

edge detection information;

line detection information;

motion detection information;

pattern recognition information;

object recognition information;

compression information; and

luminance information.

14. The method of claim 12, wherein the processed image information includes traffic signal information indicating that a yellow traffic light, a red traffic light or a stop sign is present in at least one image captured by the at least one on-board vehicle camera.

15. The method of claim 12, wherein the processed image information includes traffic speed information relating to a posted speed limit present in at least one image captured by the at least one on-board vehicle camera.

16. The method of claim 12, wherein the processed image information includes weather information based at least in part on windshield wiper operation detected in at least one image captured by the at least one on-board vehicle camera.

17. The method of claim 12, wherein the processed image information includes lane change information relating to a vehicle lane change determined from image analysis of multiple images captured by the at least one on-board vehicle camera.

18. The method of claim 12, wherein the processed image information includes proximate vehicle collision information relating to a forward proximity of the vehicle to another vehicle as determined from image analysis of multiple images captured by the at least one on-board vehicle camera.

19. The method of claim 1, wherein the at least one on-board vehicle camera is positioned to monitor a driver and/or a rear view of the vehicle, and wherein in A), the image information includes driver image information and/or rear view image information from the at least one on-board vehicle camera.

20. The method of claim 19, wherein the driver image information includes at least one of:

a first indicator corresponding to a detected driver distraction state;

a second indicator corresponding to a detected driver sleep state; and

at least one timestamp indicating when the driver image information is obtained.

21. The method of claim 1, wherein the at least one on-board vehicle camera is positioned to monitor a view from the front windshield of the vehicle, and wherein in A), the image information includes front view image information from the at least one on-board vehicle camera.

22. The method of claim 21, wherein the front view image information includes at least one of:

a number of clips captured by and available from the at least one on-board vehicle camera; and

at least one timestamp indicating when the front view image information is obtained.

23. The method of claim 1, wherein the at least one on-board vehicle camera includes a 360 degree camera system configured to monitor a 360 degree range of view inside or outside the vehicle, and wherein in A), the image information includes 360 degree view image information from the 360 degree camera system.

24. The method of claim 1, wherein the at least one on-board vehicle camera includes a video event recorder configured to record video events based at least in part on one or more exceptional forces acting on the vehicle, and wherein in A), at least some of the image information is provided by the video event recorder.

25. The method of claim 24, wherein the at least one on-board vehicle camera includes a DriveCam™

brand imaging system, and wherein in A), at least some of the image information is provided by the DriveCam™

brand imaging system.

26. The method of claim 1, wherein in C), the vehicle-related complex event includes a disobeyed stop traffic signal event, and wherein C) comprises assessing if the vehicle has run a red traffic light or a stop sign.

27. The method of claim 26, wherein:

in A), the vehicle information includes a current vehicle speed, a throttle depression reading, and/or a brake pedal application indicator;

in A), the image information comprises processed image information including traffic signal information indicating a presence of a yellow traffic light, a red traffic light or a stop sign in at least one image captured by the at least one on-board vehicle camera; and

B) comprises;

B1) detecting the current vehicle speed, the throttle depression reading, and/or the brake pedal application indicator in the at least one first input stream; and

B2) detecting the traffic signal information in the at least one second input stream.

28. The method of claim 27, wherein:

in A), the traffic signal information includes;

a first indication of a presence of a yellow traffic light in at least one first image captured by the at least one on-board vehicle camera; and

a second indication of a presence of a red traffic light in at least one second image captured by the at least one on-board vehicle camera; and

B2) comprises;

B2a) detecting the first indication; and

B2b) detecting the second indication within a predetermined time interval following B2b).

29. The method of claim 27, wherein C) comprises:

determining that the disobeyed stop signal event has occurred if;

the current vehicle speed and/or the throttle depression is approximately the same or increased upon detection in B2) of the traffic signal information and/or for a predetermined time interval following detection in B2) of the traffic signal information; and

/or

the brake pedal application indicator is absent upon detection in B2) of the traffic signal information and/or for a predetermined time interval following detection in B2) of the traffic signal information.

30. The method of claim 1, wherein in C), the vehicle-related complex event includes an excessive speed event, and wherein C) comprises assessing if a current vehicle speed exceeds a safe speed.

31. The method of claim 30, wherein:

in A), the vehicle information includes the current vehicle speed;

in A), the image information comprises processed image information including traffic speed information representing a posted speed limit present in at least one image captured by the at least one on-board vehicle camera; and

B) comprises detecting the current vehicle speed in the at least one first input stream and detecting the traffic speed information in the at least one second input stream.

32. The method of claim 31, wherein:

in A), the vehicle information comprises vehicle readings information regarding the vehicle provided by an Electronic Control Unit (ECU) of the vehicle, wherein the vehicle readings information includes the current vehicle speed.

33. The method of claim 31, wherein:

in A), the vehicle information comprises GPS information from at least one GPS receiver associated with the vehicle, wherein the GPS information includes the current vehicle speed.

34. The method of claim 31, wherein C) comprises:

determining that the excessive speed event has occurred if the current vehicle speed detected in B) exceeds the posted speed limit represented by the traffic speed information detected in B) by a predetermined percentage of the posted speed limit.

35. The method of claim 31, wherein C) comprises:

C1) determining the safe speed based at least in part on the traffic speed information detected in B) and weather information.

36. The method of claim 35, wherein:

in A), the vehicle information further includes GPS information from at least one GPS receiver associated with the vehicle;

B) further comprises detecting current longitude and latitude coordinates of the vehicle based on the GPS information in the at least one first input stream; and

C1) comprises issuing a web service call to retrieve the weather information based on the current longitude and latitude coordinates of the vehicle detected in B).

37. The method of claim 35, wherein:

in A), the vehicle information comprises vehicle readings information regarding the vehicle provided by an Electronic Control Unit (ECU) of the vehicle, wherein the vehicle readings information includes a windshield wiper activation indicator;

B) further comprises detecting the windshield wiper activation indicator in the at least one first input stream; and

C1) comprises determining the safe speed based at least in part on the traffic speed information detected in B) and the windshield wiper activation indicator detected in B).

38. The method of claim 35, wherein:

in A), the processed image information further includes at least one indication of windshield wiper operation detected in at least one image captured by the at least one on-board vehicle camera;

B) further comprises detecting the at least one indication of windshield wiper operation based on the processed image information in the at least one second input stream; and

C1) comprises determining the safe speed based at least in part on the traffic speed information detected in B) and the at least one indication of windshield wiper operation detected in B).

39. The method of claim 35, wherein C1) comprises:

if the weather information indicates at least one inclement weather condition, calculating the safe speed as a percentage of the posted speed limit represented by the traffic speed information detected in B).

40. The method of claim 31, wherein C) comprises:

C1) determining the safe speed based at least in part on the traffic speed information detected in B) and traffic information.

41. The method of claim 35, wherein:

in A), the vehicle information further includes GPS information from at least one GPS receiver associated with the vehicle;

B) further comprises detecting current longitude and latitude coordinates of the vehicle based on the GPS information in the at least one first input stream; and

C1) comprises issuing a web service call to retrieve the traffic information based on the current longitude and latitude coordinates of the vehicle detected in B).

42. The method of claim 40, wherein C1) comprises:

if the traffic information indicates a slow traffic or congested traffic condition, calculating the safe speed as a percentage of the posted speed limit represented by the traffic speed information detected in B).

43. The method of claim 1, wherein in C), the vehicle-related complex event includes a reckless driving event.

44. The method of claim 43, wherein C) comprises:

assessing if an excessive speed event has occurred for a predetermined time period.

45. The method of claim 43, wherein C) comprises:

determining a number of times that an excessive speed event has occurred during a predetermined time period.

46. The method of claim 43, wherein C) comprises:

determining a number of times that a disobeyed stop traffic signal event has occurred during a predetermined time period.

47. The method of claim 43, wherein C) comprises:

C1) assessing if an excessive speed event has occurred;

C2) assessing if a disobeyed stop traffic signal event has occurred; and

C3) determining that the reckless driving event has occurred if the excessive speed event occurs within a predetermined time interval of the disobeyed traffic signal event.

48. The method of claim 43, further comprising assessing a priority level of the reckless driving event.

49. At least one computer-readable medium encoded with a processor-executable instructions that, when executed by at least one hardware computer processor, implement a complex event processing (CEP) engine that performs a method comprising:

A) receiving at least two input streams comprising;

at least one first input stream onto which vehicle information related to a vehicle is published; and

at least one second input stream onto which image information is published, wherein the image information is based at least in part on data provided by at least one on-board vehicle camera;

B) applying at least one query to the at least one first input stream and the at least one second input stream;

C) assessing, based on B), if a vehicle-related complex event has occurred; and

D) if it is determined in C) that the vehicle-related complex event has occurred, publishing an indication of occurrence of the vehicle-related complex event on at least one output stream of the CEP engine.

50. A complex event processing (CEP) engine to assess occurrences of vehicle-related complex events, the CEP engine comprising:

at least one hardware computer processor;

at least one input/output (I/O) interface; and

at least one tangible memory to store processor-executable instructions,

wherein, upon execution of the processor-executable instructions by the at least one hardware computer processor, the CEP engine;

A) receives, via the at least one I/O interface, at least two input streams comprising;

at least one first input stream onto which vehicle information related to a vehicle is published; and

at least one second input stream onto which image information is published, wherein the image information is based at least in part on data provided by at least one on-board vehicle camera;

B) applies at least one query to the at least one first input stream and the at least one second input stream;

C) assesses, based on B), if a vehicle-related complex event has occurred; and

D) if it is determined in C) that the vehicle-related complex event has occurred, publishes, via the at least one I/O interface, an indication of occurrence of the vehicle-related complex event on at least one output stream of the CEP engine.

51. A system, comprising:

the CEP engine of claim 50;

the at least one on-board vehicle camera; and

the vehicle,

wherein the CEP engine is disposed on the vehicle.

52. The system of claim 51, wherein the at least one on-board vehicle camera includes a 360 degree camera system configured to monitor a 360 degree range of view inside or outside the vehicle, and wherein in A), the image information includes 360 degree view image information from the 360 degree camera system.

53. The system of claim 51, wherein the at least one on-board vehicle camera includes a video event recorder configured to record video events based at least in part on one or more exceptional forces acting on the vehicle, and wherein in A), at least some of the image information is provided by the video event recorder.

54. The system of claim 53, wherein the at least one on-board vehicle camera includes a DriveCam™

brand imaging system, and wherein in A), at least some of the image information is provided by the DriveCam™

brand imaging system.

55. The system of claim 51, further comprising:

an Electronic Control Unit (ECU) disposed on the vehicle and coupled to the CEP engine, wherein the vehicle information published on the at least one first input stream and received by the CEP engine comprises vehicle readings information provided by the ECU.

56. The system of claim 55, further comprising at least one environmental and/or operational sensor disposed on the vehicle and coupled to the CEP engine, wherein the at least one environmental and/or operational sensor includes at least one of:

a GPS receiver;

at least one accelerometer;

at least one humidity sensor;

at least one in-vehicle temperature sensor;

at least one external temperature sensor;

at least one light sensor; and

at least one ignition voltage sensor.

57. The system of claim 55, further comprising at least one communications interface, coupled to the CEP engine, to facilitate at least one cellular connection and/or at least one WiFi connection to the CEP engine.