Enhanced traffic detection by fusing multiple sensor data

• US 10,388,153 B1
• Filed: 06/04/2019
• Issued: 08/20/2019
• Est. Priority Date: 08/24/2018
• Status: Active Grant

• Current Assignee: Iteris Incorporated
• Original Assignee: Iteris Incorporated
• Inventors: Swaminathan, Dilip, Jain, Manu P., Gao, Yan, Kreter, Todd W., Whiting, Michael T.
• Primary Examiner(s): McNally, Kerri L
• Application Number:
  US16/430,432
• Time in Patent Office: 77 Days
• Field of Search: 340933
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/91 for traffic control G01S13/93 takes precedence
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/0125 Traffic data processing
• CPC Class Codes: G08G 1/015 with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/042 using inductive or magnetic detectors

Abstract

A framework for precision traffic analysis combines traffic sensor data from multiple sensor types, combining the strengths of each sensor type within various conditions in an intersection or roadway in which traffic activity occurs. The framework calibrates coordinate systems in images taken of the same area by multiple sensors, so that fields of view from one sensor system are transposed onto fields of view from other sensor systems to fuse the images taken into a combined detection zone, and so that objects are properly detected and classified for enhanced traffic signal control.

Claims

1. A method, comprising:

receiving input data representing fields of view of a plurality of roadway approaches at or near a traffic intersection, the input data including data collected from a first sensor, and data collected from a second sensor, for each roadway approach in the plurality of roadway approaches;

analyzing the input data within a computing environment n one or more data processing modules executed in conjunction with at least one specifically-configured processor, the one or more data processing modules configured to fuse the data from the first sensor with the data from the second sensor for each roadway approach to detect one or more objects in each field of view in the plurality of roadway approaches, and analyze input data collected for each roadway approach in a traffic network comprising one or more traffic intersections, by

at each traffic intersection in the traffic network, identifying a first object detection zone in each field of view from the data collected from the first sensor, and a second object detection zone in each field of view from the data collected from the second sensor,

selecting a plurality of boundary points in the first object detection zone, and a plurality of points in the second object detection zone that correspond to the boundary points in the first detection zone,

estimating a transformation matrix for a planar area comprising each field of view by mapping the plurality of points in a plane of each roadway approach as seen by the second sensor, to the plurality of boundary points in a plane of each roadway approach as seen by the first sensor to correct a perspective distortion between the data collected by the first sensor and the data collected by the second sensor to transpose the first object detection zone onto the second object detection zone to create a combined detection zone for each roadway approach, and

evaluating one or more attributes of each combined detection zone to detect a presence of the one or more objects at each roadway approach of each traffic intersection;

calibrating each combined detection zone from each roadway approach for each traffic intersection, and calibrating the combined detection zones in the traffic network to fuse the input data for the traffic network;

determining an adjusted signal timing for at least one roadway approach in the traffic network based on the one or more objects detected; and

generating an output signal to a desired traffic signal controller in the traffic network to implement the adjusted signal timing for the at least one roadway approach.

2. The method of claim 1, further comprising generating, as output data, a confirmation of a presence of the object in each field of view, or a confirmation of an absence of the object in each field of view.

3. The method of claim 1, wherein the first sensor is configured to capture data representing a first field of view at each roadway approach, and the second sensor is configured to capture a second field of view that is different from the first field of view.

4. The method of claim 3, wherein the first sensor is a radar sensor, and the second sensor is at least one of a video camera and a thermal camera.

5. The method of claim 1, further comprising applying the selected points that correspond to the plurality of boundary points to calibrate arbitrary planar coordinates between the first object detection zone and the second object detection zone in each roadway approach, and mapping a plurality of vectors representing common data points to transform the arbitrary planar coordinates to find an equivalent zone location and an equivalent shape between the first object detection zone and the second object detection zone in each roadway approach.

6. The method of claim 1, further comprising classifying the one or more objects in each field of view for each roadway approach as one or more of a bicycle, a truck, a passenger vehicle, a pedestrian, and an incident.

7. The method of claim 1, wherein the evaluating one or more attributes further comprises analyzing one or more images in each combined traffic detection zone to identify the one or more objects in each field of view, in one or more of an optical flow analysis to evaluate motion in each combined traffic detection zone, an edge detection analysis to evaluate spatial and temporal attributes of pixels in the input data, and a template matching analysis to compare the object with attributes of known objects.

8. The method of claim 1, further comprising analyzing polar coordinates in the data collected by the first sensor, by converting a range angle and an azimuth angle for each selected point in the data collected by the first sensor into planar coordinates to calibrate two-dimensional Cartesian coordinates with corresponding coordinates in the data collected by the second sensor.

9. The method of claim 1, wherein additional input data is captured by one or more additional sensors in or near each roadway approach, the one or more additional sensors including at least one of a magnetometer, an inductive loop, a light-based sensor, and an acoustic sensor.

10. The method of claim 1, further comprising providing the output data to a traffic management tool configured to enable a user to select a size and location of one or more of the fields of view, the first object detection zone, and the second object detection zone, and to enable a sensor configuration for each of the first sensor and the second sensor.

11. A system, comprising:

a computing environment including at least one non-transitory computer-readable storage medium having program instructions stored therein and a computer processor operable to execute the program instructions within one or more data processing modules configured to fuse input data representing fields of view of a plurality of roadway approaches at or near a traffic intersection, the input data including data collected from a first sensor, and data collected from a second sensor, for each roadway approach in the plurality of roadway approaches, the one or more data processing modules including;

one or more data processing modules configured to

identify a traffic network comprising one or more traffic intersections,

at each traffic intersection in the traffic network, identify a first object detection zone in each field of view from the data collected from the first sensor, and a second object detection zone in each field of view from the data collected from the second sensor,

select a plurality of boundary points in the first object detection zone, and a plurality of points in the second object detection zone that correspond to the plurality of boundary points in the first detection zone,

estimate a transformation matrix for a planar area comprising the each field of view of each roadway approach, by mapping the plurality of points in a plane of each roadway approach as seen by the second sensor, to the plurality of boundary points in a plane of each roadway approach as seen by the first sensor, to correct a perspective distortion between the data collected by the first sensor and the data collected by the second sensor to transpose the first object detection zone onto the second object detection zone to create a combined detection zone,

evaluate one or more attributes of each combined detection zone to detect a presence of the one or more objects at each roadway approach of each traffic intersection, and

calibrate each combined detection zone from each roadway approach for each traffic intersection, and calibrate the combined detection zones in the traffic network to fuse the input data for the traffic network; and

a traffic signal module configured to determine an adjusted signal timing for at least one roadway approach in the traffic network based on the one or more objects detected and generate an output signal to a desired traffic signal controller in the traffic network to implement the adjusted signal timing for the at least one roadway approach.

12. The system of claim 11, further comprising an output module configured to generate, as output data, a confirmation of a presence of the one or more objects in each field of view, or a confirmation of an absence of the one or more objects in each field of view.

13. The system of claim 11, wherein the first sensor is configured to capture data representing a first field of view at each roadway approach, and the second sensor is configured to capture a second field of view that is different from the first field of view.

14. The system of claim 13, wherein the first sensor is a radar sensor, and the second sensor is at least one of a video camera and a thermal camera.

15. The system of claim 11, wherein the one or more data processing modules are further configured to apply the selected points that correspond to the at least four boundary points to calibrate arbitrary planar coordinates between the first object detection zone and the second object detection zone, and mapping a plurality of vectors representing common data points to linearly transform the arbitrary planar coordinates to find an equivalent zone location and an equivalent shape between the first object detection zone and the second object detection zone.

16. The system of claim 11, wherein the one or more data processing modules are further configured to classify the one or more objects in each field of view for each roadway approach as one or more of a bicycle, a truck, a passenger vehicle, a pedestrian, and an incident.

17. The system of claim 11, wherein the evaluating one or more attributes further comprises analyzing one or more images in the combined traffic detection zone to identify the one or more objects in each field of view, in one or more of an optical flow analysis to evaluate motion in the combined traffic detection zone, an edge detection analysis to evaluate spatial and temporal attributes of pixels in the input data, and a template matching analysis to compare the object with attributes of known objects.

18. The system of claim 11, further comprising analyzing polar coordinates in the data collected by the first sensor, by converting a range angle and an azimuth angle for each selected point in the data collected by the first sensor into planar coordinates to calibrate two-dimensional Cartesian coordinates with corresponding coordinates in the data collected by the second sensor.

19. The system of claim 11, wherein additional input data is captured by one or more additional sensors in or near the roadway, the one or more additional sensors including at least one of a magnetometer, a light-based sensor, an inductive loop, and an acoustic sensor.

20. The system of claim 11, wherein the output data to a traffic management tool configured to enable a user to select a size and location of one or more of the fields of view, the first object detection zone, and the second object detection zone, and to enable a sensor configuration for each of the first sensor and the second sensor.

21. A method of detecting one or more objects in a traffic detection zone using a plurality of different sensors for adjusting traffic signal timing in a traffic network, comprising:

defining a traffic network comprised of one or more traffic intersections, each traffic intersection in the one or more traffic intersections having a plurality of roadway approaches;

for each roadway approach at the one or more traffic intersections, configuring a first object detection zone in a field of view of each roadway approach from data collected from a first sensor, and a second object detection zone in the field of view of each roadway approach from data collected from a second sensor;

calibrating arbitrary planar coordinates between the first object detection zone and the second object detection zone for each roadway approach to create a combined detection zone for the fields of view of each roadway approach, by

selecting a plurality of boundary points in the first object detection zone, and a plurality of points in the second object detection zone that correspond to the plurality of boundary points in the first object detection zone,

correcting a perspective distortion in the fields of view between the data collected by the first sensor and the data collected by the second sensor from the selected plurality of boundary points, by estimating a transformation matrix for a planar area comprising the field of view to map the plurality of points in a plane of each roadway approach as seen by the second sensor, to the plurality of boundary points in a plane of each roadway approach as seen by the first sensor,

finding an equivalent zone location and an equivalent shape between the first detection zone and the second detection zone,

detecting one or more objects at each roadway approach by analyzing one or more attributes of the combined detection zone;

fusing data for the traffic network, by calibrating each combined detection zone from each roadway approach for each traffic intersection, and calibrating the combined detection zones of the one or more traffic intersections;

adjusting a signal timing for at least one roadway approach in the traffic network based on the one or more objects detected; and

communicating with a desired traffic signal controller in the traffic network to implement an adjusted signal timing for the at least one roadway approach.

22. The method of claim 21, further comprising generating, as output data, a confirmation of a presence of the one or more objects in each field of view, or a confirmation of an absence of the one or more objects in each field of view.

23. The method of claim 21, wherein the first sensor is configured to capture data representing a first field of view at each roadway approach, and the second sensor is configured to capture a second field of view that is different from the first field of view.

24. The method of claim 23, wherein the first sensor is a radar sensor, and the second sensor is at least one of a video camera and a thermal camera.

25. The method of claim 21, further comprising classifying the one or more objects in each field of view for each roadway approach as one or more of a bicycle, a truck, a passenger vehicle, a pedestrian, and an incident.

26. The method of claim 21, further comprising mapping a plurality vectors representing common data points to linearly transform the arbitrary polar coordinates, and transposing the plurality of vectors to fuse the first detection zone and the second detection zone into the combined traffic detection zone.

27. The method of claim 21, wherein the detecting one or more objects at each roadway approach further comprises analyzing one or more images in the combined traffic detection zone to identify the one or more objects in each field of view, in one or more of an optical flow analysis to evaluate motion in the combined traffic detection zone, an edge detection analysis to evaluate spatial and temporal attributes of pixels in the input data, and a template matching analysis to compare the object with attributes of known objects.

28. The method of claim 21, further comprising analyzing polar coordinates in the data collected by the first sensor, by converting a range angle and an azimuth angle for each selected point in the data collected by the first sensor into planar coordinates.

29. The method of claim 21, wherein additional input data is captured by one or more additional sensors in or near each roadway approach, the one or more additional sensors including at least one of a magnetometer, a light-based sensor an inductive loop, and an acoustic sensor.

30. The method of claim 21, further comprising providing the output data to a traffic management tool configured to enable a user to select a size and location of one or more of the fields of view, the first object detection zone, and the second object detection zone, and to enable a sensor configuration for each of the first sensor and the second sensor.