Distributed system for mining, correlating, and analyzing locally obtained traffic data including video

• US 10,438,071 B2
• Filed: 01/25/2017
• Issued: 10/08/2019
• Est. Priority Date: 01/25/2017
• Status: Active Grant

• Current Assignee: Echelon Corporation (Renesas Electronics Corporation)
• Original Assignee: Echelon Corporation (Renesas Electronics Corporation)
• Inventors: Modi, Sohrab, Singh, Shalender
• Primary Examiner(s): Beck, Leron
• Application Number:
  US15/414,892
• Publication Number:
  US 20180211116A1
• Time in Patent Office: 986 Days
• Field of Search: 348149
• CPC Class Codes: G06F 16/75 Clustering; Classification
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/625 License plates
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G08G 1/0116 from roadside infrastructure, e.g. beacons
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 for classifying traffic situation
• CPC Class Codes: G08G 1/015 with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 84/042 Public Land Mobile systems, e.g. cellular systems
• CPC Class Codes: H04W 84/12 WLAN [Wireless Local Area Networks]

Abstract

A distributed system is disclosed for mining, correlating and analyzing locally obtained traffic video data. The system comprises a central controller configured to correlate traffic counting meta data gathered at a thoroughfare, to transmit control information to specify one or more traffic counting events to be monitored by at least one video unit located at the thoroughfare and to receive traffic counting meta data from the at least one video unit. The at least one video unit located at the thoroughfare, is configured to process a video stream from a video camera while monitoring the traffic counting events at the thoroughfare specified by the central controller, to analyze and evaluate the traffic counting event, to encode traffic meta data characterizing the analysis, and to transmit the meta data to the central controller. Further meta data can be acquired by mining the meta data history or any captured raw video footage.

Claims

1. A distributed system for correlating and analyzing locally obtained traffic video data, comprising:

a central controller configured to distribute a specification of types of vehicular traffic events to monitor, to a plurality of video units located along a thoroughfare;

at least a sub-plurality of the plurality of video units, in response to receiving the specification of vehicular traffic events to monitor, each configured to activate a respective video camera to monitor for the specified vehicular traffic events, analyze with a first algorithm, a video stream of monitored vehicular traffic events produced by the respective video camera, encode meta data representing the analyzed monitored vehicular traffic events, and transmit the meta data to the central controller;

at least one video unit of the plurality of video units, in response to receiving the specification of vehicular traffic events to monitor, configured to activate a respective video camera of the at least one video unit to monitor for the specified vehicular traffic events, and analyze with the first algorithm, a video stream of monitored vehicular traffic events produced by the respective video camera of the at least one video unit;

the at least one video unit of the plurality configured to determine that the analyzed vehicular traffic event requires a different algorithm for analysis, re-process with the different algorithm, the video stream of monitored vehicular traffic events produced by the respective video camera of the at least one video unit, encode re-processed meta data, and transmit the re-processed meta data to the central controller;

a first one of the plurality of video units located at a first location along the thoroughfare and a second one of the plurality of video units located at a different location along the thoroughfare and monitoring the same event as the first one of the plurality of video units, with the second one of the plurality of video units feeding its meta data to the first one of the plurality of video units, the first one of the plurality of video units, outputting the combined meta data representing the same event, and sending the combined meta data to the central controller; and

the central controller further configured to correlate the meta data and re-processed meta data representing the monitored vehicular traffic events received from each of the sub-plurality of the plurality of video units and the at least one video unit of the plurality of video units, in monitoring a vehicular traffic event.

2. The system of claim 1, wherein the video units are activated by respective sensors that sense vehicular traffic, to activate the respective video cameras to search for and monitor the specified vehicular traffic events.

3. The system of claim 1, wherein the video units perform image recognition of the video stream produced by the respective video camera, to recognize at least one of a vehicle type, manufacturer, color, features such as a moon roof, damage on the vehicle, license plate number or an identifying sticker displayed on the vehicle.

4. The system of claim 1, wherein the meta data includes geographic coordinates of each transmitting video unit and a time stamp indicating a time of occurrence of the monitored event.

5. The system of claim 1, wherein the central controller correlates the received traffic monitoring meta data based on an identified vehicle characteristic represented in the meta data, including at least one of vehicle type, manufacturer, color, features such as a moon roof, damage on the vehicle, license plate number, or an identifying sticker displayed on the vehicle.

6. The system of claim 1, wherein the at least one video unit of the plurality is configured to perform a loop to repetitively re-process the video stream of monitored vehicular traffic events produced by the respective video camera, repetitively encode the re-processed meta data and transmit the repetitively re-processed meta data to the central controller.

7. The system of claim 1, wherein the central controller is configured to perform a loop to repetitively re-evaluate the received meta data of monitored vehicular traffic events produced by the respective video units.

8. The system of claim 1, wherein the at least one video unit of the plurality is configured to perform data mining to access and evaluate previous traffic counting data stored in a historical database, to identify patterns in traffic events related to at least one of road capacities, road congestion related to weather conditions, day of the week, delays due to road maintenance or collisions, encoding the identified patterns as meta data and transmitting the meta data to the central controller.

9. The system of claim 1, wherein the central controller is configured to perform data mining to access and evaluate previous traffic counting data stored in a historical database, to identify patterns in traffic events related to at least one of road capacities, road congestion related to weather conditions, day of the week, delays due to road maintenance or collisions.

10. The system of claim 1, further comprising:

a communications unit in each of the video units being configured to transmit low bandwidth meta data to the central controller over a communications medium selected from the group consisting of twisted pair, coax cable, Ethernet, Infrared, RFID, WiFi, Bluetooth, Bluetooth Low Energy, ultra-narrow band communications protocol from Sigfox, LTE-M, Low Power Wireless Area Network (LPWAN) protocol, any M2M communication protocol, cellular, IEEE 802.15.4 RF, or LoRa Low Power Wide Area Network;

the communications unit in the central controller being configured to receive the low bandwidth meta data from each of the video units over the communications medium.

11. The system of claim 1, further comprising:

the communications unit in each of the video units being a power line communications unit configured to transmit low bandwidth meta data over installed power lines to the central controller;

the communications unit in the central controller being a power line communications unit, configured to receive the low bandwidth meta data over installed power lines from each of the video units; and

the power line communications unit in the central controller further configured to transmit or receive data communications over installed power lines to electrically powered devices in the thoroughfare where power is needed and control desired.

12. The system of claim 1, further comprising:

a sensor located at each of the video units, configured to sense distance and motion of a vehicle, each of the video units, configured to prepare low bandwidth meta data representing the vehicle'"'"'s position and motion sensed by the sensor; and

the central controller further configured to receive the low bandwidth meta data and to track the vehicle as it moves in the thoroughfare.

13. The system of claim 1, further comprising:

a sensor coupled to each of the video units, configured to view the thoroughfare and to detect an approach of a vehicle and transmit a signal to the video unit, indicating the approaching vehicle;

each of the video units configured to capture an identification of the approaching vehicle in response to the signal from the sensor and to transmit the identification to the central controller, the identification being based on at least one of a license plate number or an identifying sticker displayed on the vehicle or model and make of vehicle or color of vehicle.

21. The system of claim 1, further comprising:

a first one of the plurality of video units located at a first location along the thoroughfare and a second one of the plurality of video units located at a different location along the thoroughfare and monitoring the same event as the first one of the plurality of video units, with the second one of the plurality of video units feeding its video stream to the first one of the plurality of video units, the first one of the plurality of video units, analyzing the two video streams representing two views of the same event, outputting combined meta data representing the same event, and sending the combined meta data to the central controller.

22. The system of claim 1, further comprising:

at least one of the plurality of video units, including inference engine logic configured to analyze the vehicular traffic events and to encode the meta data characterizing the analysis of the vehicular traffic events;

wherein the meta data output by the inference engine logic includes at least one of a color based search of objects after an event has occurred or a search for tracks of objects as they move through a scene based on location, color, or classification of the object; and

the central controller further configured to correlate the meta data representing the monitored vehicular traffic events received from each of the sub-plurality of the plurality of video units and the at least one video unit of the plurality of video units, in monitoring a vehicular traffic event.

23. The distributed system of claim 22, further comprising:

the at least one video unit of the plurality of video units, configured to perform image recognition of the video stream, to recognize vehicle characteristics, analyze the recognized vehicle characteristics, and trigger additional video analysis algorithms to gather more information, including gauging vehicle directions and speeds after a collision event is detected.

24. The distributed system of claim 23, further comprising:

the at least one video unit of the plurality of video units, configured to perform data mining to access and evaluate previous traffic counting data stored in a historical database, to identify patterns in traffic events related to at least one of road capacities, road congestion related to weather conditions, day of the week, delays due to road maintenance or collisions, encode the identified patterns as meta data and transmit the meta data to the central controller.

14. A distributed system for correlating and analyzing locally obtained traffic video data, comprising:

a central controller, including a processor and memory including computer program code configured to cause the controller to correlate traffic counting meta data gathered at a thoroughfare, the controller further including a communications unit configured to transmit the control information to specify one or more traffic counting events to be monitored by a plurality of video units located along the thoroughfare and to receive the traffic counting meta data from at least one of the plurality of video units;

the at least one video unit located at the thoroughfare, including a video camera, video frame processing logic, a processor and memory including computer program code configured to cause the video frame processing logic to process a video stream from the video camera while monitoring the traffic counting events at the thoroughfare specified by the central controller, to identify a traffic counting event associated with the thoroughfare, and to analyze the traffic counting event with a first algorithm;

the at least one video unit further configured to determine that the analyzed traffic counting event requires a different algorithm for analysis, re-process the traffic counting event with the different algorithm, and encode re-processed meta data, the at least one video unit including a communications unit configured to transmit the re-processed meta data characterizing analysis of the traffic counting event; and

a first one of the plurality of video units located at a first location along the thoroughfare and the at least one video unit located at a different location along the thoroughfare and monitoring the same event as the first one of the plurality of video units, with the at least one video unit feeding its meta data to the first one of the plurality of video units, the first one of the plurality of video units, outputting the combined meta data representing the same event, and sending the combined meta data to the central controller.

15. The system of claim 14, further comprising:

the communications unit in the at least one video unit being configured to transmit the low bandwidth event message to the central controller over a communications medium selected from the group consisting of twisted pair, coax cable, Ethernet, Infrared, RFID, WiFi, Bluetooth, Bluetooth Low Energy, ultra-narrow band communications protocol from Sigfox, LTE-M, Low Power Wireless Area Network (LPWAN) protocol, any M2M communication protocol, cellular, IEEE 802.15.4 RF, or LoRa Low Power Wide Area Network;

the communications unit in the central controller being configured to receive the low bandwidth event message from the at least one video unit over the communications medium.

16. The system of claim 14, further comprising:

the communications unit in the at least one video unit being a power line communications unit configured to transmit a low bandwidth event message over installed power lines to the central controller;

the communications unit in the central controller being a power line communications unit, configured to receive the low bandwidth event message over installed power lines from the at least one video unit; and

the power line communications unit in the central controller further configured to transmit or receive data communications over installed power lines to electrically powered devices in the thoroughfare where power is needed and control desired.

17. The system of claim 14, further comprising:

a sensor located at the at least one video unit, configured to sense distance and motion of a vehicle, the at least one video unit, configured to prepare a low bandwidth event message representing the vehicle'"'"'s position and motion sensed by the sensor; and

the central controller further configured to receive the low bandwidth event message and to track the vehicle as it moves in the thoroughfare.

18. The system of claim 14, further comprising:

a sensor coupled to the at least one video unit, is configured to view the thoroughfare and to detect an approach of a vehicle and transmit a signal to the at the at least one video unit, indicating the approaching vehicle;

the at the at least one video unit is configured to capture an identification of the approaching vehicle in response to the signal from the sensor and to transmit the identification to the central controller, the identification being based on at least one of a license plate number or an identifying sticker displayed on the vehicle or model and make of vehicle or color of vehicle.

19. A method for correlating and analyzing locally obtained traffic video data, comprising:

distributing, by a central controller configured, a specification of types of vehicular traffic events to monitor, to a plurality of video units located along a thoroughfare;

activating, at least a sub-plurality of the plurality of video units, in response to receiving the specification of vehicular traffic events to monitor, a respective video camera to monitor for the specified vehicular traffic events, and analyzing with a first algorithm a video stream of monitored vehicular traffic events produced by the respective video camera, encoding meta data representing the analyzed monitored vehicular traffic events, and transmitting the encoded meta data to the central controller;

at least one video unit of the plurality of video units, in response to receiving the specification of vehicular traffic events to monitor, activating a respective video camera of the at least one video unit to monitor for the specified vehicular traffic events, and analyzing with the first algorithm, a video stream of monitored vehicular traffic events produced by the respective video camera of the at least one video unit;

the at least one video unit of the plurality determining that the analyzed vehicular traffic event requires a different algorithm for analysis, re-processing with the different algorithm, the video stream of monitored vehicular traffic events produced by the respective video camera of the at least one video unit, encoding re-processed meta data, and transmitting the re-processed meta data;

monitoring the same event by a first one of the plurality of video units located at a first location along the thoroughfare and a second one of the plurality of video units located at a different location along the thoroughfare, with the second one of the plurality of video units feeding its meta data to the first one of the plurality of video units, the first one of the plurality of video units, outputting the combined meta data representing the same event, and sending the combined meta data to the central controller; and

correlating, by the central controller, the meta data representing the analyzed monitored vehicular traffic events received from each of the sub-plurality of the plurality of video units of video units, in monitoring a vehicular traffic event.

20. A method for correlating and analyzing locally obtained traffic video data, comprising:

processing a video stream, by each of a plurality of video units, while monitoring traffic counting events along a thoroughfare specified by a central controller, to identify a traffic counting event associated with the thoroughfare, to analyze the traffic counting event with a first algorithm;

monitoring the same event by a first one of the plurality of video units located at a first location along the thoroughfare and a second one of the plurality of video units located at a different location along the thoroughfare;

determining, by the second one of the plurality of video units, that the analyzed traffic counting event requires a different algorithm for analysis, re-processing with the different algorithm, the video stream of monitored traffic counting events produced by the respective video camera of the second one of the plurality of video units, encoding re-processed meta data, and transmitting the re-processed meta data;

the second one of the plurality of video units feeding its meta data to the first one of the plurality of video units, the first one of the plurality of video units, outputting the combined meta data representing the same event, and sending the combined meta data to the central controller; and

receiving, by the central controller, the re-processed meta data characterizing analysis of the traffic counting event, and evaluating the re-processed traffic counting meta data.