System and method for vehicle occlusion detection

• US 10,783,381 B2
• Filed: 05/19/2019
• Issued: 09/22/2020
• Est. Priority Date: 08/31/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Yu, Hongkai, Yan, Zhipeng, Wang, Panqu, Chen, Pengfei
• Primary Examiner(s): Brushaber, Frederick M
• Application Number:
  US16/416,248
• Publication Number:
  US 20190272433A1
• Time in Patent Office: 492 Days
• CPC Class Codes: B60W 2050/0088 Adaptive recalibration
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 60/00272 relying on extrapolation of current movement
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/89 for mapping or imaging
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/7753 Incorporation of unlabelled data, e.g. multiple instance learning [MIL]
• CPC Class Codes: G06V 10/87 using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A system and method for vehicle occlusion detection is disclosed. A particular embodiment includes: receiving training image data from a training image data collection system; obtaining ground truth data corresponding to the training image data; performing a training phase to train a plurality of classifiers, a first classifier being trained for processing static images of the training image data, a second classifier being trained for processing image sequences of the training image data; receiving image data from an image data collection system associated with an autonomous vehicle; and performing an operational phase including performing feature extraction on the image data, determining a presence of an extracted feature instance in multiple image frames of the image data by tracing the extracted feature instance back to a previous plurality of N frames relative to a current frame, applying the first trained classifier to the extracted feature instance if the extracted feature instance cannot be determined to be present in multiple image frames of the image data, and applying the second trained classifier to the extracted feature instance if the extracted feature instance can be determined to be present in multiple image frames of the image data.

Claims

1. A system comprising:

a data processor; and

an autonomous vehicle occlusion detection system, executable by the data processor, the autonomous vehicle occlusion detection system being configured to perform an autonomous vehicle occlusion detection operation for autonomous vehicles, the autonomous vehicle occlusion detection operation being configured to;

receive image data from an image data collection system associated with an autonomous vehicle; and

perform an operational phase including performing feature extraction on the image data, determine a presence of an extracted feature instance in multiple image frames of the image data by tracing the extracted feature instance back to a previous plurality of N frames relative to a current frame, apply a first trained classifier to the extracted feature instance if the extracted feature instance cannot be determined to be present in multiple image frames of the image data, and apply a second trained classifier to the extracted feature instance if the extracted feature instance can be determined to be present in multiple image frames of the image data.

2. The system of claim 1 being further configured to receive training image data from a training image data collection system;

obtain ground truth data corresponding to the training image data; and

perform a training phase to train the first classifier for processing static images of the training image data, and to train the second classifier for processing image sequences of the training image data.

3. The system of claim 2 wherein the training phase being configured to obtain ground truth data including labeling data and object relationship information for the training image data, the object relationship information including a status for each of the objects in a frame of the training image data, the status including a state from the group consisting of:

1) occluding another object;

(2) occluded by another object;

(3) there is no overlap with another object, and (4) in between two objects.

4. The system of claim 1 being configured to associate a plurality of feature dimensions of quantity F with each extracted feature instance processed by the first classifier.

5. The system of claim 1 being configured to associate a plurality of feature dimensions of quantity F*N with each extracted feature instance processed by the second classifier.

6. The system of claim 1 being configured to apply a bounding box to extracted features of the image data, the bounding box being partitioned into a plurality of portions.

7. The system of claim 1 being configured to generate object-level contour detections for each extracted feature of the image data.

8. A computer-implemented vehicle occlusion detection method comprising:

receiving image data from an image data collection system associated with an autonomous vehicle; and

performing an operational phase using an in-vehicle data processor installed in the autonomous vehicle, the operational phase including using the in-vehicle data processor to perform feature extraction on the image data, using the in-vehicle data processor to determine a presence of an extracted feature instance in multiple image frames of the image data by tracing the extracted feature instance back to a previous plurality of N frames relative to a current frame, using the in-vehicle data processor to apply a first trained classifier to the extracted feature instance if the extracted feature instance cannot be determined to be present in multiple image frames of the image data, and using the in-vehicle data processor to apply a second trained classifier to the extracted feature instance if the extracted feature instance can be determined to be present in multiple image frames of the image data.

9. The method of claim 8 including receiving training image data from a training image data collection system;

obtaining ground truth data corresponding to the training image data; and

performing a training phase to train the first classifier for processing static images of the training image data, and to train the second classifier for processing image sequences of the training image data.

10. The method of claim 9 wherein the training phase includes obtaining ground truth data including labeling data and object relationship information for the training image data, the object relationship information including a status for each of the objects in a frame of the training image data, the status including a state from the group consisting of:

1) occluding another object;

(2) occluded by another object;

(3) there is no overlap with another object, and (4) in between two objects.

11. The method of claim 8 including associating a plurality of feature dimensions of quantity F with each extracted feature instance processed by the first classifier.

12. The method of claim 8 including associating a plurality of feature dimensions of quantity F*N with each extracted feature instance processed by the second classifier.

13. The method of claim 8 including applying a bounding box to extracted features of the image data, the bounding box being partitioned into a plurality of portions.

14. The method of claim 8 including generating object-level contour detections for each extracted feature of the image data.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive image data from an image data collection system associated with an autonomous vehicle; and

perform an operational phase including performing feature extraction on the image data, determine a presence of an extracted feature instance in multiple image frames of the image data by tracing the extracted feature instance back to a previous plurality of N frames relative to a current frame, apply a first trained classifier to the extracted feature instance if the extracted feature instance cannot be determined to be present in multiple image frames of the image data, and apply a second trained classifier to the extracted feature instance if the extracted feature instance can be determined to be present in multiple image frames of the image data.

16. The non-transitory machine-useable storage medium of claim 15 being further configured to receive training image data from a training image data collection system;

obtain ground truth data corresponding to the training image data; and

perform a training phase to train the first classifier for processing static images of the training image data, and to train the second classifier for processing image sequences of the training image data.

17. The non-transitory machine-useable storage medium of claim 16 wherein the training phase being configured to obtain ground truth data including labeling data and object relationship information for the training image data, the object relationship information including a status for each of the objects in a frame of the training image data, the status including a state from the group consisting of:

1) occluding another object;

(2) occluded by another object;

(3) there is no overlap with another object, and (4) in between two objects.

18. The non-transitory machine-useable storage medium of claim 15 being configured to associate a plurality of feature dimensions of quantity F with each extracted feature instance processed by the first classifier.

19. The non-transitory machine-useable storage medium of claim 15 being configured to associate a plurality of feature dimensions of quantity F*N with each extracted feature instance processed by the second classifier.

20. The non-transitory machine-useable storage medium of claim 15 wherein the first classifier is a static classifier and the second classifier is a temporal classifier.