APPARATUS AND METHOD FOR PROCESSING DATA OF HETEROGENEOUS SENSORS IN INTEGRATED MANNER TO CLASSIFY OBJECTS ON ROAD AND DETECT LOCATIONS OF OBJECTS

• US 20130307981A1
• Filed: 11/21/2012
• Published: 11/21/2013
• Est. Priority Date: 05/15/2012
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kwak, Dong-Yong, Jang, Jeong-Ah
• Granted Patent: US 9,154,741 B2
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08G 1/015 with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Disclosed herein are an apparatus and method for processing the data of heterogeneous sensors in an integrated manner to classify one or more objects on a road and detect the locations of the objects. The apparatus includes a camera image processing unit, a scanning information processing unit, and an integrated processing unit. The camera image processing unit generates the results of the processing of camera images based on the camera images collected from one or more cameras. The scanning information processing unit generates the results of the processing of scanning information based on the scanning information collected from one or more laser scanners. The integrated processing unit generates the types and final locations of the objects located on the road by processing the generated results of the processing of the camera images and the generated results of the processing of the scanning information in an integrated manner.

Claims

1. An apparatus for processing data of heterogeneous sensors in an integrated manner to classify one or more objects on a road and detect locations of the objects, the apparatus comprising:

a camera image processing unit configured to generate results of processing of camera images based on the camera images collected from one or more cameras installed on the road;

a scanning information processing unit configured to generate results of processing of scanning information based on the scanning information collected from one or more laser scanners installed on the road; and

an integrated processing unit configured to generate types and final locations of the objects located on the road by processing the generated results of the processing of the camera images and the generated results of the processing of the scanning information in an integrated manner.

2. The apparatus of claim 1, wherein the camera image processing unit:

detects a length of a sensible road based on a height and angle of each of the installed cameras included in previously stored camera information; and

detects a road area and a non-road area based on variations in color between pixels of the received camera images and then detects a width of the road.

3. The apparatus of claim 1, wherein the camera image processing unit:

detects types of objects located on the road based on previously stored reference objects and pixel groups of the camera images, and sets an object type ID for each of the objects based on the detected types of objects;

converts the objects detected from the camera images into rectangles, and detects a location in which diagonals of each of the resulting rectangles intersect each other as an object center location of a corresponding object; and

generates the results of the processing of the camera images including object type IDs and object center locations of the objects located on the road.

4. The apparatus of claim 1, wherein the scanning information processing unit detects estimated locations of the objects located on the road by performing multiple data processing on location information included in laser scanner information associated with sensor numbers of the laser scanners and the received scanning information.

5. The apparatus of claim 1, wherein the scanning information processing unit:

generates a road sensing map based on a direction vector of motion of each of the objects depending on a test drive or an initial driving situation of a vehicle over a predetermined period of time or based on previously stored road information;

classifies the objects located on the road as one or more moving objects and/or one or more fixed objects based on data monitoring of the road area based on the generated road sensing map, and detects sizes and a distribution of sensing points of the moving objects and the fixed objects; and

performs an object identity test based on sizes and a distribution of the detected sensing points, sets object IDs, and groups the objects based on the set object IDs.

6. The apparatus of claim 1, wherein the scanning information processing unit generates the results of the processing of the scanning information including object IDs, types of grouping, and estimated locations.

7. The apparatus of claim 1, wherein the integrated processing unit:

maps the objects on the road to an integrated object sensing map composed of grid cells based on the results of the processing of the camera images and the results of the processing of the scanning information;

calculates a period and probability of each of grid cells of the integrated object sensing map, to which the objects have been mapped, being occupied;

if the calculated probability of being occupied is equal to or higher than a set threshold value, determines an identity for each grid cell by comparing an object type ID of an object occupying the grid cell with an object ID of the object, and, if the identity for each grid cell is normal, sets an object type ID for the grid cell; and

sets an average representative location of estimated locations of the results of processing of the laser scanner-related information as a final location of a corresponding object.

8. The apparatus of claim 1, wherein the integrated processing unit generates results of the integrated processing including collection times, object type IDs and object locations.

9. The apparatus of claim 1, further comprising:

a storage unit configured to store heterogeneous sensor information including at least one of a sensor type, location, height and angle of each of the installed cameras and laser scanners, and a road number on which each of the installed cameras and laser scanners is installed; and

road information including at least one of a road number, a start node, an end node, and a number of traffic lanes.

10. The apparatus of claim 1, further comprising a transmission unit configured to send results of the integrated processing generated by the integrated processing unit to a vehicle or one or more other information servers.

11. A method of processing data of heterogeneous sensors in an integrated manner to classify one or more objects on a road and detect locations of the objects, the method comprising:

generating, by a camera image processing unit, results of processing of camera images based on the camera images collected from one or more cameras installed on the road;

generating, by a scanning information processing unit, results of processing of scanning information based on the scanning information collected from one or more laser scanners installed on the road; and

generating, by an integrated processing unit, types and final locations of the objects located on the road by processing the generated results of the processing of the camera images and the generated results of the processing of the scanning information in an integrated manner.

12. The method of claim 11, wherein the generating the results of the processing of the camera images comprises:

extracting, by the camera image processing unit, a road area based on camera information detected from a storage unit and variations in color between pixels of the camera images;

extracting, by the camera image processing unit, the objects from the camera images based on reference objects stored in the storage unit;

detecting, by the camera image processing unit, the types of objects on the road based on previously stored reference objects and pixel groups of the camera images, and then setting, by the camera image processing unit, object type IDs for the objects;

converting, by the camera image processing unit, the extracted objects into rectangles, and then extracting, by the camera image processing unit, a location at which diagonals of each of the rectangles intersect each other as an object center location of a corresponding object; and

generating, by the camera image processing unit, the results of the processing of the camera images including the set object type IDs and object center locations.

13. The method of claim 12, wherein the extracting a road area comprises:

detecting, by the camera image processing unit, camera information including a height and angle of each of the installed cameras from the storage unit;

detecting, by the camera image processing unit, a length of a sensible road based on the detected camera information;

detecting, by the camera image processing unit, a road area and a non-road area based on variations in color between pixels of the received camera images; and

detecting, by the camera image processing unit, a width of the road based on the detected road area and non-road area.

14. The method of claim 11, wherein the generating results of the processing of the scanning information comprises:

detecting, by the scanning information processing unit, estimated locations of the objects on the road by performing multiple data processing on laser scanner information and the scanning information;

generating, by the scanning information processing unit, a road sensing map based on a direction vector of motion of each of the objects depending on a test drive or an initial driving situation of a vehicle over a predetermined period of time or based on previously stored road information;

performing, by the scanning information processing unit, an object identity test based on sizes and a distribution of sensing points of the objects mapped to the generated road sensing map;

grouping, by the scanning information processing unit, objects based on results of the object identity test; and

generating, by the scanning information processing unit, the results of the processing of the scanning information including object IDs, types of grouping, and the estimated locations.

15. The method of claim 14, wherein:

the performing an object identity test comprises setting, by the scanning information processing unit, a same object ID for same objects; and

the grouping the objects comprises grouping, by the scanning information processing unit, the objects for which the same object ID has been set as a group.

16. The method of claim 11, wherein the generating types and final locations of the objects comprises:

mapping, by the integrated processing unit, the received results of the processing of the camera images and the received results of the processing of the scanning information to an integrated object sensing map;

calculating, by the integrated processing unit, a period and probability of each of grid cells of the integrated object sensing map being occupied;

determining, by the integrated processing unit, an identity for each grid cell based on the calculated probability of the grid cell being occupied;

if the identity for each grid cell is normal, classifying, by the integrated processing unit, the objects for the corresponding grid cells;

estimating, by the integrated processing unit, locations of the classified objects; and

generating, by the integrated processing unit, results of the integrated processing.

17. The method of claim 16, wherein the determining an identity for each grid cell comprises:

comparing, by the integrated processing unit, the calculated probability of being occupied with a set threshold value;

if the probability of being occupied is equal to or higher than the set threshold value, determining, by the integrated processing unit, the identity for each grid cell by comparing an object type ID of an object occupying the corresponding grid cell with an object ID of the object; and

if the identity for each grid cell is normal, setting, by the integrated processing unit, an object type ID for the corresponding grid cell.

18. The method of claim 16, wherein the estimating locations of the objects comprises setting, by the integrated processing unit, an average representative location of estimated locations of the results of processing of the laser scanner-related information as a final location of a corresponding object.

19. The method of claim 16, wherein the generating the results of the integrated processing comprises generating, by the integrated processing unit, results of the integrated processing including collection times, object type IDs and object locations.

20. The method of claim 11, further comprising sending, by a transmission unit, results of the integrated processing generated by the integrated processing unit to a vehicle or one or more other information servers.