DETECTING ROADWAY OBJECTS IN REAL-TIME IMAGES

• US 20160267331A1
• Filed: 03/12/2015
• Published: 09/15/2016
• Est. Priority Date: 03/12/2015
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Pillai, Preeti Jayagopi, Yalla, Veeraganesh, Oguchi, Kentaro, Nomoto, Hirokazu
• Granted Patent: US 9,916,508 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60T 2201/08 Lane monitoring; Lane Keeping Systems
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

The disclosure includes a method that receives a real-time image of a road from a camera sensor communicatively coupled to an onboard computer of a vehicle. The method includes dividing the real-time image into superpixels. The method includes merging the superpixels to form superpixel regions. The method includes generating prior maps from a dataset of road scene images. The method includes drawing a set of bounding boxes where each bounding box surrounds one of the superpixel regions. The method includes comparing the bounding boxes in the set of bounding boxes to a road prior map to identify a road region in the real-time image. The method includes pruning bounding boxes from the set of bounding boxes to reduce the set to remaining bounding boxes. The method may include using a categorization module that identifies the presence of a road scene object in the remaining bounding boxes.

Claims

1. A method for identifying roadway objects for a vehicle, the method comprising:

determining, using a camera sensor communicatively coupled to an onboard computer of a vehicle, a real-time image of a road;

dividing the real-time image into superpixels;

merging the superpixels to form superpixel regions based on similarity of the superpixels;

generating prior maps from a dataset of road scene images, the prior maps including a road prior map;

drawing a set of bounding boxes where each bounding box surrounds one of the superpixel regions;

comparing bounding boxes in the set of bounding boxes to the road prior map to identify a road region in the real-time image;

pruning bounding boxes from the set of bounding boxes that are outside the road region to reduce the set to remaining bounding boxes; and

performing feature extraction and categorization to identify objects within the set of remaining bounding boxes.

2. The method of claim 1, further comprising providing route guidance based on the identified objects.

3. The method claim of 1, wherein merging the superpixels to form superpixel regions based on similarity of the superpixels includes determining texture and similarity of colors of the superpixels.

4. The method of claim 1, further comprising:

determining a box area relative to an image resolution of the real-time image for each of the bounding boxes in the set of bounding boxes; and

pruning bounding boxes from the set of bounding boxes with box areas that fall below a threshold value to reduce the set to the remaining bounding boxes.

5. The method of claim 1, further comprising:

determining a bounding box aspect ratio and a location of each bounding box in the real-time image based on proximity to the road region;

comparing the bounding box aspect ratio to a list of approved bounding box aspect ratios for the location based on the prior maps; and

pruning bounding boxes from the set of bounding boxes where the bounding box aspect ratio fails to match one of the approved bounding box aspect ratios in the list of approved bounding box aspect ratios for the location to reduce the set to the remaining bounding boxes.

6. The method of claim 1, further comprising:

determining a bounding box aspect ratio and a location of each bounding box in the real-time image based on proximity to the road region, wherein performing feature extraction and categorization comprises applying a type of categorization model to each of the remaining bounding boxes based on the bounding box aspect ratio and the location of each bounding box;

identifying a presence or an absence of an object within each of the bounding boxes in the set of remaining bounding boxes; and

determining a type of object based on the feature extraction and categorization.

7. The method of claim 6, wherein the type of categorization model includes one or more of a vehicle categorization model, a pedestrian categorization model, a pole categorization model, a bicycle categorization model, a motorcycle categorization model, a stationary bicycle and motorcycle categorization model, and a construction cone categorization model.

8. The method of claim 1, further comprising:

determining coordinates and a centroid value for each of the bounding boxes in the set of bounding boxes;

comparing the superpixel regions to the road prior map to identify a horizon line and an approximate vanishing point in the real-time image; and

pruning bounding boxes from the set of bounding boxes with a centroid value located above the horizon line.

9. The method of claim 1, wherein merging the superpixels to form superpixel regions based on similarity of the superpixels is based on using a graph-based agglomerative technique.

10. The method of claim 1, wherein the superpixel regions include structures and objects.

11. The method of claim 1, wherein the prior maps include locations for objects and structures where the objects and structures include one or more of a vehicle, a construction cone, a pedestrian, a bicycle, a sky, a motorcycle, foliage, a tree, an electrical pole, a streetlight, a road, and a road sign.

12. The method of claim 1, further comprising preprocessing the real-time image to remove noise and downsample.

13. A non-transitory computer-readable medium having computer instructions stored thereon that are executable by a processing device to perform or control performance of steps comprising:

determining, using a camera sensor communicatively coupled to an onboard computer of a vehicle, a real-time image of a road;

dividing the real-time image into superpixels;

merging the superpixels to form superpixel regions based on similarity of the superpixels;

generating prior maps from a dataset of road scene images, the prior maps including a road prior map;

drawing a set of bounding boxes where each bounding box surrounds one of the superpixel regions;

comparing bounding boxes in the set of bounding boxes to the road prior map to identify a road region in the real-time image;

pruning bounding boxes from the set of bounding boxes that are outside the road region to reduce the set to remaining bounding boxes; and

performing feature extraction and categorization to identify objects within the set of remaining bounding boxes.

14. The non-transitory computer-readable medium of claim 13, the steps further comprising providing route guidance based on the identified objects.

15. The non-transitory computer-readable medium of claim 13, wherein merging the superpixels to form superpixel regions based on similarity of the superpixels includes determining texture and similarity of colors of the superpixels.

16. The non-transitory computer-readable medium of claim 13, the steps further comprising:

determining a box area relative to an image resolution of the real-time image for each of the bounding boxes in the set of bounding boxes; and

pruning bounding boxes from the set of bounding boxes with box areas that fall below a threshold value to reduce the set to the remaining bounding boxes.

17. The non-transitory computer-readable medium of claim 13, the steps further comprising

determining a bounding box aspect ratio and a location of each bounding box in the real-time image based on proximity to the road region;

comparing the bounding box aspect ratio to a list of approved bounding box aspect ratios for the location based on the prior maps; and

pruning bounding boxes from the set of bounding boxes where the bounding box aspect ratio fails to match one of the approved bounding box aspect ratios in the list of approved bounding box aspect ratios for the location to reduce the set to the remaining bounding boxes.

18. The non-transitory computer-readable medium of claim 13, the steps further comprising:

determining a bounding box aspect ratio and a location of each bounding box in the real-time image based on proximity to the road region, wherein performing feature extraction and categorization comprises applying a type of categorization model to each of the remaining bounding boxes based on the bounding box aspect ratio and the location of each bounding box;

identifying a presence or an absence of an object within each of the bounding boxes in the set of remaining bounding boxes; and

determining a type of object based on the feature extraction and categorization.

19. The non-transitory computer-readable medium of claim 18, wherein the type of categorization model includes one or more of a vehicle categorization model, a pedestrian categorization model, a pole categorization model, a bicycle categorization model, a motorcycle categorization model, a stationary bicycle and motorcycle categorization model, and a construction cone categorization model.

20. A method for identifying roadway objects, the method comprising:

determining, using a camera sensor communicatively coupled to an onboard computer of a vehicle, a real-time image of a road;

dividing the real-time image into superpixels;

merging the superpixels to form superpixel regions based on similarity of the superpixels;

generating prior maps from a dataset of road scene images, the prior maps including a road prior map;

drawing a set of bounding boxes where each bounding box surrounds one of the superpixel regions;

comparing bounding boxes in the set of bounding boxes to the road prior map to identify a road region in the real-time image;

determining a bounding box aspect ratio and a location of each bounding box in the real-time image based on proximity to the road region;

comparing the bounding box aspect ratio to a list of approved bounding box aspect ratios for the location based on the prior maps;

pruning bounding boxes from the set of bounding boxes;

where the bounding box aspect ratio fails to match one of the approved bounding box aspect ratios in the list of approved bounding box aspect ratios for the location outside the road region to reduce the set to remaining bounding boxes, or

the bounding boxes are outside the road region;

performing feature extraction and categorization to identify objects within the remaining bounding boxes by assigning a type of categorization to each of the remaining bounding boxes based on the bounding box aspect ratio and the location of each bounding box;

identifying a presence or an absence of an object within each of the remaining bounding boxes; and

determining a type of object based on the feature extraction and categorization.