Real-time detection of traffic situation

• US 10,540,554 B2
• Filed: 03/29/2018
• Issued: 01/21/2020
• Est. Priority Date: 03/29/2018
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Guo, Rui, Oguchi, Kentaro
• Primary Examiner(s): Liew, Alex Kok S
• Application Number:
  US15/939,308
• Publication Number:
  US 20190303686A1
• Time in Patent Office: 663 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

A method extracts a region of interest (ROI) from an image depicting a roadway; executes a first object detector that detects, in the ROI, a first situation object matching a first object type; generates a first object confidence score for the first situation object; executes, in parallel with the first object detector, a second object detector that detects, in the ROI, a second situation object matching a second object type different from the first object type; generates a second object confidence score for the second situation object; determines that the first object type and the second object type are associated with a prospective situation category; combines the first object confidence score and the second object confidence score into a situation confidence score corresponding to the prospective situation category; evaluates the situation confidence score to determine that a traffic situation defined by the prospective situation category is depicted in the ROI.

Claims

1. A method comprising:

extracting a region of interest (ROI) from an image depicting a roadway;

executing a first object detector that detects, in the ROI, a first situation object matching a first object type based on object features of the first situation object, the first situation object reflecting a first physical roadway object depicted in the ROI;

generating a first object confidence score for the first situation object based on a level of similarity between the first situation object and the first object type;

executing, in parallel with the first object detector, a second object detector that detects, in the ROI, a second situation object matching a second object type that is different from the first object type based on object features of the second situation object, the second situation object reflecting a second physical roadway object depicted in the ROI;

generating a second object confidence score for the second situation object based on a level of similarity between the second situation object and the second object type;

determining that the first object type detected by the first object detector and the second object type detected by the second object detector are associated with a first prospective situation category;

combining the first object confidence score and the second object confidence score into a first situation confidence score corresponding to the first prospective situation category; and

evaluating the first situation confidence score to determine that the first prospective situation category applies to the ROI and that a first traffic situation defined by the first prospective situation category is depicted in the ROI.

2. The method of claim 1, further comprising:

determining a first bounding box indicating the first situation object in the ROI;

determining a second bounding box indicating the second situation object in the ROI; and

aggregating the first bounding box indicating the first situation object and the second bounding box indicating the second situation object to generate a situation region of the first traffic situation depicted in the ROI.

3. The method of claim 1, wherein extracting the ROI from the image includes:

extracting, from the image, the ROI associated with a prominent color.

4. The method of claim 1, wherein:

the first prospective situation category is associated with a first set of object types in a data store, the first set of object types including the first object type detected by the first object detector and the second object type detected by the second object detector; and

a second prospective situation category is associated with a second set of object types in the data store, the second set of object types including a third object type detected by a third object detector and a fourth object type detected by a fourth object detector, the third object type being different from the fourth object type.

5. The method of claim 1, further comprising:

executing, in parallel with the first object detector, a third object detector that detects, in the ROI, a third situation object matching a third object type based on object features of the third situation object, the third situation object reflecting a third physical roadway object depicted in the ROI;

generating a third object confidence score for the third situation object based on a level of similarity between the third situation object and the third object type;

executing, in parallel with the first object detector, a fourth object detector that detects, in the ROI, a fourth situation object matching a fourth object type based on object features of the fourth situation object, the fourth situation object reflecting a fourth physical roadway object depicted in the ROI;

generating a fourth object confidence score for the fourth situation object based on a level of similarity between the fourth situation object and the fourth object type;

determining that the third object type detected by the third object detector and the fourth object type detected by the fourth object detector are associated with a second prospective situation category;

combining the third object confidence score and the fourth object confidence score into a second situation confidence score corresponding to the second prospective situation category; and

evaluating the first situation confidence score and the second situation confidence score to determine that the first prospective situation category applies to the ROI, the second prospective situation category applies to the ROI, and that the first traffic situation defined by the first prospective situation category and a second traffic situation defined by the second prospective situation category are depicted in the ROI.

6. The method of claim 1, further comprising:

executing, in parallel with the first object detector, a third object detector that detects, in the ROI, a third situation object matching a third object type based on object features of the third situation object, the third situation object reflecting a third physical roadway object depicted in the ROI;

generating a third object confidence score for the third situation object based on a level of similarity between the third situation object and the third object type;

executing, in parallel with the first object detector, a fourth object detector that detects, in the ROI, a fourth situation object matching a fourth object type based on object features of the fourth situation object, the fourth situation object reflecting a fourth physical roadway object depicted in the ROI;

generating a fourth object confidence score for the fourth situation object based on a level of similarity between the fourth situation object and the fourth object type;

determining that the third object type detected by the third object detector and the fourth object type detected by the fourth object detector are associated with a second prospective situation category;

combining the third object confidence score and the fourth object confidence score into a second situation confidence score corresponding to the second prospective situation category; and

evaluating the first situation confidence score and the second situation confidence score to determine that the first prospective situation category applies to the ROI, the second prospective situation category does not apply to the ROI, and that the first traffic situation defined by the first prospective situation category is depicted in the ROI.

7. The method of claim 1, further comprising:

executing, in parallel with the first object detector, a third object detector that detects, in the ROI, a third situation object matching a third object type based on object features of the third situation object, the third situation object reflecting a third physical roadway object depicted in the ROI;

generating a third object confidence score for the third situation object based on a level of similarity between the third situation object and the third object type;

determining that the first object type detected by the first object detector and the third object type detected by the third object detector are associated with a second prospective situation category;

combining the first object confidence score and the third object confidence score into a second situation confidence score corresponding to the second prospective situation category; and

evaluating the first situation confidence score and the second situation confidence score to determine that the first prospective situation category applies to the ROI, the second prospective situation category does not apply to the ROI, and that the first traffic situation defined by the first prospective situation category is depicted in the ROI.

8. The method of claim 1, wherein combining the first object confidence score and the second object confidence score into the first situation confidence score corresponding to the first prospective situation category includes:

retrieving a first situation detection weight of the first object type in detecting traffic situations defined by the first prospective situation category;

retrieving a second situation detection weight of the second object type in detecting the traffic situations defined by the first prospective situation category; and

computing the first situation confidence score based on the first object confidence score of the first situation object, the first situation detection weight of the first object type, the second object confidence score of the second situation object, and the second situation detection weight of the second object type.

9. The method of claim 8, further comprising:

determining that a second traffic situation defined by the first prospective situation category is depicted in a training image; and

adjusting one or more of the first situation detection weight of the first object type in detecting the traffic situations defined by the first prospective situation category and the second situation detection weight of the second object type in detecting the traffic situations defined by the first prospective situation category based on a comparison of the second traffic situation detected in the training image to a predetermined target output associated with the training image.

10. A system comprising:

one or more processors;

one or more memories storing instructions that, when executed by the one or more processors, cause the system to;

extract a region of interest (ROI) from an image depicting a roadway;

execute a first object detector that detects, in the ROI, a first situation object matching a first object type based on object features of the first situation object, the first situation object reflecting a first physical roadway object depicted in the ROI;

generate a first object confidence score for the first situation object based on a level of similarity between the first situation object and the first object type;

execute, in parallel with the first object detector, a second object detector that detects, in the ROI, a second situation object matching a second object type that is different from the first object type based on object features of the second situation object, the second situation object reflecting a second physical roadway object depicted in the ROI;

generate a second object confidence score for the second situation object based on a level of similarity between the second situation object and the second object type;

determine that the first object type detected by the first object detector and the second object type detected by the second object detector are associated with a first prospective situation category;

combine the first object confidence score and the second object confidence score into a first situation confidence score corresponding to the first prospective situation category; and

evaluate the first situation confidence score to determine that the first prospective situation category applies to the ROI and that a first traffic situation defined by the first prospective situation category is depicted in the ROI.

11. The system of claim 10, wherein the instructions, when executed by the one or more processors, further cause the system to:

determine a first bounding box indicating the first situation object in the ROI;

determine a second bounding box indicating the second situation object in the ROI; and

aggregate the first bounding box indicating the first situation object and the second bounding box indicating the second situation object to generate a situation region of the first traffic situation depicted in the ROI.

12. The system of claim 10, wherein to extract the ROI from the image includes:

extracting, from the image, the ROI associated with a prominent color.

13. The system of claim 10, wherein:

the first prospective situation category is associated with a first set of object types in a data store, the first set of object types including the first object type detected by the first object detector and the second object type detected by the second object detector; and

a second prospective situation category is associated with a second set of object types in the data store, the second set of object types including a third object type detected by a third object detector and a fourth object type detected by a fourth object detector, the third object type being different from the fourth object type.

14. The system of claim 10, wherein the instructions, when executed by the one or more processors, further cause the system to:

execute, in parallel with the first object detector, a third object detector that detects, in the ROI, a third situation object matching a third object type based on object features of the third situation object, the third situation object reflecting a third physical roadway object depicted in the ROI;

generate a third object confidence score for the third situation object based on a level of similarity between the third situation object and the third object type;

execute, in parallel with the first object detector, a fourth object detector that detects, in the ROI, a fourth situation object matching a fourth object type based on object features of the fourth situation object, the fourth situation object reflecting a fourth physical roadway object depicted in the ROI;

generate a fourth object confidence score for the fourth situation object based on a level of similarity between the fourth situation object and the fourth object type;

determine that the third object type detected by the third object detector and the fourth object type detected by the fourth object detector are associated with a second prospective situation category;

combine the third object confidence score and the fourth object confidence score into a second situation confidence score corresponding to the second prospective situation category; and

evaluate the first situation confidence score and the second situation confidence score to determine that the first prospective situation category applies to the ROI, the second prospective situation category applies to the ROI, and that the first traffic situation defined by the first prospective situation category and a second traffic situation defined by the second prospective situation category are depicted in the ROI.

15. The system of claim 10, wherein the instructions, when executed by the one or more processors, further cause the system to:

execute, in parallel with the first object detector, a third object detector that detects, in the ROI, a third situation object matching a third object type based on object features of the third situation object, the third situation object reflecting a third physical roadway object depicted in the ROI;

generate a third object confidence score for the third situation object based on a level of similarity between the third situation object and the third object type;

execute, in parallel with the first object detector, a fourth object detector that detects, in the ROI, a fourth situation object matching a fourth object type based on object features of the fourth situation object, the fourth situation object reflecting a fourth physical roadway object depicted in the ROI;

generate a fourth object confidence score for the fourth situation object based on a level of similarity between the fourth situation object and the fourth object type;

determine that the third object type detected by the third object detector and the fourth object type detected by the fourth object detector are associated with a second prospective situation category;

combine the third object confidence score and the fourth object confidence score into a second situation confidence score corresponding to the second prospective situation category; and

evaluate the first situation confidence score and the second situation confidence score to determine that the first prospective situation category applies to the ROI, the second prospective situation category does not apply to the ROI, and that the first traffic situation defined by the first prospective situation category is depicted in the ROI.

16. The system of claim 10, wherein the instructions, when executed by the one or more processors, further cause the system to:

execute, in parallel with the first object detector, a third object detector that detects, in the ROI, a third situation object matching a third object type based on object features of the third situation object, the third situation object reflecting a third physical roadway object depicted in the ROI;

generate a third object confidence score for the third situation object based on a level of similarity between the third situation object and the third object type;

determine that the first object type detected by the first object detector and the third object type detected by the third object detector are associated with a second prospective situation category;

combine the first object confidence score and the third object confidence score into a second situation confidence score corresponding to the second prospective situation category; and

evaluate the first situation confidence score and the second situation confidence score to determine that the first prospective situation category applies to the ROI, the second prospective situation category does not apply to the ROI, and that the first traffic situation defined by the first prospective situation category is depicted in the ROI.

17. The system of claim 10, wherein to combine the first object confidence score and the second object confidence score into the first situation confidence score corresponding to the first prospective situation category includes:

retrieving a first situation detection weight of the first object type in detecting traffic situations defined by the first prospective situation category;

retrieving a second situation detection weight of the second object type in detecting the traffic situations defined by the first prospective situation category; and

computing the first situation confidence score based on the first object confidence score of the first situation object, the first situation detection weight of the first object type, the second object confidence score of the second situation object, and the second situation detection weight of the second object type.

18. The system of claim 17, wherein the instructions, when executed by the one or more processors, further cause the system to:

determine that a second traffic situation defined by the first prospective situation category is depicted in a training image; and

adjust one or more of the first situation detection weight of the first object type in detecting the traffic situations defined by the first prospective situation category and the second situation detection weight of the second object type in detecting the traffic situations defined by the first prospective situation category based on a comparison of the second traffic situation detected in the training image to a predetermined target output associated with the training image.

19. A method comprising:

extracting a first region of interest (ROI) from a first image depicting a roadway;

executing, in parallel, a first object detector that detects a first situation object matching a first object type in the first ROI and a second object detector that detects a second situation object matching a second object type in the first ROI, the first object type being different from the second object type;

generating a first object confidence score for the first situation object and a second object confidence score for the second situation object;

determining that the first object type detected by the first object detector and the second object type detected by the second object detector are associated with a first prospective situation category;

combining the first object confidence score and the second object confidence score into a first situation confidence score corresponding to the first prospective situation category;

evaluating the first situation confidence score to determine that the first prospective situation category does not apply to the first ROI of the first image and that a first traffic situation defined by the first prospective situation category is not depicted in the first ROI of the first image;

extracting a second ROI from a second image depicting the roadway;

executing, in parallel, the first object detector that detects a third situation object matching the first object type in the second ROI and the second object detector that detects a fourth situation object matching the second object type in the second ROI;

generating a third object confidence score for the third situation object and a fourth object confidence score for the fourth situation object;

combining the third object confidence score and the fourth object confidence score into a second situation confidence score corresponding to the first prospective situation category; and

evaluating the second situation confidence score to determine that the first prospective situation category applies to the second ROI of the second image and that the first traffic situation defined by the first prospective situation category is depicted in the second ROI of the second image.