Enhanced traffic detection by fusing multiple sensor data

• US 10,140,855 B1
• Filed: 08/24/2018
• Issued: 11/27/2018
• Est. Priority Date: 08/24/2018
• Status: Active Grant

• Current Assignee: Iteris Incorporated
• Original Assignee: Iteris Incorporated
• Inventors: Swaminathan, Dilip, Jain, Manu P., Gao, Yan, Kreter, Todd W., Whiting, Michael T.
• Primary Examiner(s): McNally, Kerri L
• Application Number:
  US16/112,072
• Time in Patent Office: 95 Days
• Field of Search: 340933
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/91 for traffic control G01S13/93 takes precedence
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/0125 Traffic data processing
• CPC Class Codes: G08G 1/015 with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/042 using inductive or magnetic detectors

Abstract

A framework for precision traffic analysis combines traffic sensor data from multiple sensor types, combining the strengths of each sensor type within various conditions in an intersection or roadway in which traffic activity occurs. The framework calibrates coordinate systems in images taken of the same area by multiple sensors, so that fields of view from one sensor system are transposed onto fields of view from other sensor systems to fuse the images taken into a combined detection zone, and so that objects are properly detected and classified for enhanced traffic signal control.

Claims

1. A method, comprising:

receiving input data representing a field of view of a roadway at or near a traffic intersection, the input data including data collected from a first sensor, and data collected from a second sensor;

analyzing the input data within a computing environment in one or more data processing modules executed in conjunction with at least one specifically-configured processor, the one or more data processing modules configured to fuse the data from the first sensor with the data from the second sensor, by

identifying a first object detection zone in the field of view from the data collected from the first sensor, and a second object detection zone in the field of view from the data collected from the second sensor,

selecting a plurality of boundary points in the first object detection zone, and a plurality of points in the second object detection zone that correspond to the boundary points in the first detection zone,

estimating a transformation matrix for a planar area comprising the field of view by mapping the plurality of points in a plane of the road roadway as seen by the second sensor, to the plurality of boundary points in a plane of the roadway as seen by the first sensor to correct a perspective distortion between the data collected by the first sensor and the data collected by the second sensor to transpose the first object detection zone onto the second object detection zone to create a combined detection zone, by applying the selected points that correspond to the plurality of boundary points to calibrate arbitrary planar coordinates between the first object detection zone and the second object detection zone, and mapping a plurality of vectors representing common data points to transform the arbitrary planar coordinates to find an equivalent zone location and an equivalent shape between the first object detection zone and the second object detection zone; and

evaluating one or more attributes of the combined detection zone to detect an object in the field of view of the roadway.

2. The method of claim 1, further comprising generating, as output data, a confirmation of a presence of the object in the field of view, or a confirmation of an absence of the object in the field of view.

3. The method of claim 1, wherein the first sensor is configured to capture data representing a first field of view of the traffic intersection, and the second sensor is configured to capture a second field of view that is different from the first field of view.

4. The method of claim 3, wherein the first sensor is a radar sensor, and the second sensor is at least one of a video camera and a thermal camera.

5. The method of claim 1, further comprising classifying the one or more objects in the field of view of the roadway as one or more of a bicycle, a truck, a passenger vehicle, a pedestrian, and an incident.

6. The method of claim 1, wherein the evaluating one or more attributes further comprises analyzing one or more images in the combined traffic detection zone to identify the object in the field of view, in one or more of an optical flow analysis to evaluate motion in the combined traffic detection zone, an edge detection analysis to evaluate spatial and temporal attributes of pixels in the input data, and a template matching analysis to compare the object with attributes of known objects.

7. The method of claim 1, further comprising analyzing polar coordinates in the data collected by the first sensor, by converting a range angle and an azimuth angle for each selected point in the data collected by the first sensor into planar coordinates to calibrate two-dimensional Cartesian coordinates with corresponding coordinates in the data collected by the second sensor.

8. The method of claim 1, wherein additional input data is captured by one or more additional sensors in or near the roadway, the one or more additional sensors including at least one of a magnetometer, an inductive loop, a light-based sensor, and an acoustic sensor.

9. The method of claim 1, further comprising providing the output data to a traffic management tool configured to enable a user to select a size and location of one or more of the field of view, the first object detection zone, and the second object detection zone, and to enable a sensor configuration for each of the first sensor and the second sensor.

10. A system, comprising:

a computing environment including at least one non-transitory computer-readable storage medium having program instructions stored therein and a computer processor operable to execute the program instructions within one or more data processing modules configured to fuse input data representing a field of view of a roadway at or near a traffic intersection, the input data including data collected from a first sensor, and data collected from a second sensor, the one or more data processing modules including;

one or more data processing modules configured to

identify a first object detection zone in the field of view from the data collected from the first sensor, and a second object detection zone in the field of view from the data collected from the second sensor,

select a plurality of boundary points in the first object detection zone, and a plurality of points in the second object detection zone that correspond to the plurality of boundary points in the first detection zone,

estimate a transformation matrix for a planar area comprising the field of view, by mapping the plurality of points in a plane of the road roadway as seen by the second sensor, to the plurality of boundary points in a plane of the roadway as seen by the first sensor, to correct a perspective distortion between the data collected by the first sensor and the data collected by the second sensor to transpose the first object detection zone onto the second object detection zone to create a combined detection zone, by applying the selected points that correspond to the at least four boundary points to calibrate arbitrary planar coordinates between the first object detection zone and the second object detection zone, and mapping a plurality of vectors representing common data points to linearly transform the arbitrary planar coordinates to find an equivalent zone location and an equivalent shape between the first object detection zone and the second object detection zone; and

an object detection module configured to evaluate one or more attributes of the combined detection zone to detect an object in the field of view of the roadway.

11. The system of claim 10, further comprising an output module configured to generate, as output data, a confirmation of a presence of the object in the field of view, or a confirmation of an absence of the object in the field of view.

12. The system of claim 10, wherein the first sensor is configured to capture data representing a first field of view of the traffic intersection, and the second sensor is configured to capture a second field of view that is different from the first field of view.

13. The system of claim 12, wherein the first sensor is a radar sensor, and the second sensor is at least one of a video camera and a thermal camera.

14. The system of claim 10, wherein the one or more data processing modules are further configured to classify the one or more objects in the field of view of the roadway as one or more of a bicycle, a truck, a passenger vehicle, a pedestrian, and an incident.

15. The system of claim 10, wherein the evaluating one or more attributes further comprises analyzing one or more images in the combined traffic detection zone to identify the object in the field of view, in one or more of an optical flow analysis to evaluate motion in the combined traffic detection zone, an edge detection analysis to evaluate spatial and temporal attributes of pixels in the input data, and a template matching analysis to compare the object with attributes of known objects.

16. The system of claim 10, further comprising analyzing polar coordinates in the data collected by the first sensor, by converting a range angle and an azimuth angle for each selected point in the data collected by the first sensor into planar coordinates to calibrate two-dimensional Cartesian coordinates with corresponding coordinates in the data collected by the second sensor.

17. The system of claim 10, wherein additional input data is captured by one or more additional sensors in or near the roadway, the one or more additional sensors including at least one of a magnetometer, a light-based sensor, an inductive loop, and an acoustic sensor.

18. The system of claim 10, wherein the output data to a traffic management tool configured to enable a user to select a size and location of one or more of the field of view, the first object detection zone, and the second object detection zone, and to enable a sensor configuration for each of the first sensor and the second sensor.

19. A method of detecting an object in a traffic detection zone using a plurality of different sensors, comprising:

configuring a first object detection zone in a field of view of a roadway at or near a traffic intersection from data collected from a first sensor, and a second object detection zone in the field of view of the roadway at or near the traffic intersection from data collected from a second sensor;

calibrating arbitrary planar coordinates between the first object detection zone and the second object detection zone to create a combined detection zone for the field of view, by

selecting a plurality of boundary points in the first object detection zone, and a plurality of points in the second object detection zone that correspond to the plurality of boundary points in the first object detection zone,

correcting a perspective distortion in the field of view between the data collected by the first sensor and the data collected by the second sensor from the selected plurality of boundary points, by estimating a transformation matrix for a planar area comprising the field of view to map the plurality of points in a plane of the road roadway as seen by the second sensor, to the plurality of boundary points in a plane of the roadway as seen by the first sensor,

finding an equivalent zone location and an equivalent shape between the first detection zone and the second detection zone, by mapping a plurality vectors representing common data points to linearly transform the arbitrary polar coordinates, and transposing the plurality of vectors to fuse the first detection zone and the second detection zone into the combined traffic detection zone; and

detecting an object in the roadway by analyzing one or more attributes of the combined traffic detection zone.

20. The method of claim 19, further comprising generating, as output data, a confirmation of a presence of the object in the field of view, or a confirmation of an absence of the object in the field of view.

21. The method of claim 19, wherein the first sensor is configured to capture data representing a first field of view of the traffic intersection, and the second sensor is configured to capture a second field of view that is different from the first field of view.

22. The method of claim 21, wherein the first sensor is a radar sensor, and the second sensor is at least one of a video camera and a thermal camera.

23. The method of claim 19, further comprising classifying the one or more objects in the field of view of the roadway as one or more of a bicycle, a truck, a passenger vehicle, a pedestrian, and an incident.

24. The method of claim 19, wherein the detecting an object in the roadway further comprises analyzing one or more images in the combined traffic detection zone to identify the object in the field of view, in one or more of an optical flow analysis to evaluate motion in the combined traffic detection zone, an edge detection analysis to evaluate spatial and temporal attributes of pixels in the input data, and a template matching analysis to compare the object with attributes of known objects.

25. The method of claim 19, further comprising analyzing polar coordinates in the data collected by the first sensor, by converting a range angle and an azimuth angle for each selected point in the data collected by the first sensor into planar coordinates.

26. The method of claim 19, wherein additional input data is captured by one or more additional sensors in or near the roadway, the one or more additional sensors including at least one of a magnetometer, a light-based sensor an inductive loop, and an acoustic sensor.

27. The method of claim 19, further comprising providing the output data to a traffic management tool configured to enable a user to select a size and location of one or more of the field of view, the first object detection zone, and the second object detection zone, and to enable a sensor configuration for each of the first sensor and the second sensor.