System and method for vehicle occlusion detection

• US 10,311,312 B2
• Filed: 10/28/2017
• Issued: 06/04/2019
• Est. Priority Date: 08/31/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: Tusimple
• Inventors: Yu, Hongkai, Yan, Zhipeng, Wang, Panqu, Chen, Pengfei
• Primary Examiner(s): Brushaber, Frederick M
• Application Number:
  US15/796,769
• Publication Number:
  US 20190065864A1
• Time in Patent Office: 584 Days
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A system and method for vehicle occlusion detection is disclosed. A particular embodiment includes: receiving training image data from a training image data collection system; obtaining ground truth data corresponding to the training image data; performing a training phase to train a plurality of classifiers, a first classifier being trained for processing static images of the training image data, a second classifier being trained for processing image sequences of the training image data; receiving image data from an image data collection system associated with an autonomous vehicle; and performing an operational phase including performing feature extraction on the image data, determining a presence of an extracted feature instance in multiple image frames of the image data by tracing the extracted feature instance back to a previous plurality of N frames relative to a current frame, applying the first trained classifier to the extracted feature instance if the extracted feature instance cannot be determined to be present in multiple image frames of the image data, and applying the second trained classifier to the extracted feature instance if the extracted feature instance can be determined to be present in multiple image frames of the image data.

Claims

1. A system comprising:

a data processor; and

an autonomous vehicle occlusion detection system, executable by the data processor, the autonomous vehicle occlusion detection system being configured to perform an autonomous vehicle occlusion detection operation for autonomous vehicles, the autonomous vehicle occlusion detection operation being configured to;

receive training image data from a training image data collection system;

obtain ground truth data corresponding to the training image data;

perform a training phase to train a plurality of classifiers, a first classifier being trained for processing static images of the training image data, a second classifier being trained for processing image sequences of the training image data;

receive image data from an image data collection system associated with an autonomous vehicle; and

perform an operational phase including performing feature extraction on the image data, determine a presence of an extracted feature instance in multiple image frames of the image data by tracing the extracted feature instance back to a previous plurality of N frames relative to a current frame, apply the first trained classifier to the extracted feature instance if the extracted feature instance cannot be determined to be present in multiple image frames of the image data, and apply the second trained classifier to the extracted feature instance if the extracted feature instance can be determined to be present in multiple image frames of the image data.

2. The system of claim 1 wherein the training phase being configured to obtain ground truth data from a manual image annotation or labeling process.

3. The system of claim 1 wherein the training phase being configured to obtain ground truth data including labeling data and object relationship information for the training image data, the object relationship information including a status for each of the objects in a frame of the training image data, the status including a state from the group consisting of:

1) occluding another object;

(2) occluded by another object;

(3) there is no overlap with another object, and (4) in between two objects.

4. The system of claim 1 being configured to associate a plurality of feature dimensions of quantity F with each extracted feature instance processed by the first classifier.

5. The system of claim 1 being configured to associate a plurality of feature dimensions of quantity F*N with each extracted feature instance processed by the second classifier.

6. The system of claim 1 being configured to apply a bounding box to extracted features of the image data, the bounding box being partitioned into a plurality of portions.

7. The system of claim 1 being configured to generate object-level contour detections for each extracted feature of the image data.

8. A computer-implemented vehicle occlusion detection method comprising:

receiving, by use of a data processor, training image data from a training image data collection system;

obtaining ground truth data corresponding to the training image data;

performing, by use of the data processor, a training phase to train a plurality of classifiers, a first classifier being trained for processing static images of the training image data, a second classifier being trained for processing image sequences of the training image data;

receiving image data from an image data collection system associated with an autonomous vehicle; and

performing an operational phase using an in-vehicle data processor installed in the autonomous vehicle, the operational phase including using the in-vehicle data processor to perform feature extraction on the image data, using the in-vehicle data processor to determine a presence of an extracted feature instance in multiple image frames of the image data by tracing the extracted feature instance back to a previous plurality of N frames relative to a current frame, using the in-vehicle data processor to apply the first trained classifier to the extracted feature instance if the extracted feature instance cannot be determined to be present in multiple image frames of the image data, and using the in-vehicle data processor to apply the second trained classifier to the extracted feature instance if the extracted feature instance can be determined to be present in multiple image frames of the image data.

9. The method of claim 8 wherein the training phase includes obtaining ground truth data from a manual image annotation or labeling process.

10. The method of claim 8 wherein the training phase includes obtaining ground truth data including labeling data and object relationship information for the training image data, the object relationship information including a status for each of the objects in a frame of the training image data, the status including a state from the group consisting of:

1) occluding another object;

(2) occluded by another object;

(3) there is no overlap with another object, and (4) in between two objects.

11. The method of claim 8 including associating a plurality of feature dimensions of quantity F with each extracted feature instance processed by the first classifier.

12. The method of claim 8 including associating a plurality of feature dimensions of quantity F*N with each extracted feature instance processed by the second classifier.

13. The method of claim 8 including applying a bounding box to extracted features of the image data, the bounding box being partitioned into a plurality of portions.

14. The method of claim 8 including generating object-level contour detections for each extracted feature of the image data.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive training image data from a training image data collection system;

obtain ground truth data corresponding to the training image data;

perform a training phase to train a plurality of classifiers, a first classifier being trained for processing static images of the training image data, a second classifier being trained for processing image sequences of the training image data;

receive image data from an image data collection system associated with an autonomous vehicle; and

perform an operational phase including performing feature extraction on the image data, determine a presence of an extracted feature instance in multiple image frames of the image data by tracing the extracted feature instance back to a previous plurality of N frames relative to a current frame, apply the first trained classifier to the extracted feature instance if the extracted feature instance cannot be determined to be present in multiple image frames of the image data, and apply the second trained classifier to the extracted feature instance if the extracted feature instance can be determined to be present in multiple image frames of the image data.

16. The non-transitory machine-useable storage medium of claim 15 wherein the training phase being configured to obtain ground truth data from a manual image annotation or labeling process.

17. The non-transitory machine-useable storage medium of claim 15 wherein the training phase being configured to obtain ground truth data including labeling data and object relationship information for the training image data, the object relationship information including a status for each of the objects in a frame of the training image data, the status including a state from the group consisting of:

1) occluding another object;

(2) occluded by another object;

(3) there is no overlap with another object, and (4) in between two objects.

18. The non-transitory machine-useable storage medium of claim 15 being configured to associate a plurality of feature dimensions of quantity F with each extracted feature instance processed by the first classifier.

19. The non-transitory machine-useable storage medium of claim 15 being configured to associate a plurality of feature dimensions of quantity F*N with each extracted feature instance processed by the second classifier.

20. The non-transitory machine-useable storage medium of claim 15 wherein the first classifier is a static classifier and the second classifier is a temporal classifier.