System and method for vehicle taillight state recognition

• US 10,733,465 B2
• Filed: 08/16/2019
• Issued: 08/04/2020
• Est. Priority Date: 09/20/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Wang, Panqu, Li, Tian
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US16/542,770
• Publication Number:
  US 20190370574A1
• Time in Patent Office: 354 Days
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

A system and method for taillight signal recognition using a convolutional neural network is disclosed. An example embodiment includes: receiving a plurality of image frames from one or more image-generating devices of an autonomous vehicle; using a single-frame taillight illumination status annotation dataset and a single-frame taillight mask dataset to recognize a taillight illumination status of a proximate vehicle identified in an image frame of the plurality of image frames, the single-frame taillight illumination status annotation dataset including one or more taillight illumination status conditions of a right or left vehicle taillight signal, the single-frame taillight mask dataset including annotations to isolate a taillight region of a vehicle; and using a multi-frame taillight illumination status dataset to recognize a taillight illumination status of the proximate vehicle in multiple image frames of the plurality of image frames, the multiple image frames being in temporal succession.

Claims

1. A method of recognizing a taillight signal of a vehicle, the method comprising:

receiving a plurality of image frames from one or more image-generating devices of an autonomous vehicle;

using a single-frame taillight illumination status annotation dataset and a single-frame taillight mask dataset to recognize a taillight illumination status of a proximate vehicle identified in an image frame of the plurality of image frames, the single-frame taillight illumination status annotation dataset including one or more taillight illumination status conditions of a right or left vehicle taillight signal, the single-frame taillight mask dataset including annotations to isolate a taillight region of a vehicle; and

using a multi-frame taillight illumination status dataset to recognize a taillight illumination status of the proximate vehicle in multiple image frames of the plurality of image frames, the multiple image frames being in temporal succession.

2. The method of claim 1 further including training a deep convolutional neural network with the single-frame taillight illumination status annotation dataset and using the deep convolutional neural network to perform feature extraction on the plurality of image frames.

3. The method of claim 2 wherein the feature extraction is performed using an object detection process and application of bounding boxes to define the detection of objects in the image frame.

4. The method of claim 2 including performing local feature extraction on each side of a taillight mask of the single-frame taillight mask dataset.

5. The method of claim 1 further including performing feature aggregation.

6. The method of claim 5 wherein feature aggregation includes integrating an extracted feature with its feature counterpart in a previous image frame within a predefined time window.

7. The method of claim 1 wherein the taillight illumination status includes an indication from the group consisting of:

the taillight is invisible or occluded, the taillight is visible but not illuminated or dark, the taillight is visible and illuminated or bright, and the taillight status is unknown.

8. The method of claim 1 wherein the taillight illumination status includes an inferred indication from the group consisting of:

brake, right turn on, left turn on, and hazard on.

9. The method of claim 1 further including performing an autonomous vehicle control signal or route planning modification based on the recognized taillight illumination status of the proximate vehicle.

10. A system comprising:

a data processor; and

a taillight signal recognition module, executable by the data processor, the taillight signal recognition module being configured to perform a taillight signal recognition operation using a deep convolutional neural network, the taillight signal recognition operation being configured to;

receive a plurality of image frames from one or more image-generating devices of an autonomous vehicle;

use a single-frame taillight illumination status annotation dataset and a single-frame taillight mask dataset to recognize a taillight illumination status of a proximate vehicle identified in an image frame of the plurality of image frames, the single-frame taillight illumination status annotation dataset including one or more taillight illumination status conditions of a right or left vehicle taillight signal, the single-frame taillight mask dataset including annotations to isolate a taillight region of a vehicle; and

use a multi-frame taillight illumination status dataset to recognize a taillight illumination status of the proximate vehicle in multiple image frames of the plurality of image frames, the multiple image frames being in temporal succession.

11. The system of claim 10 being further configured to train the deep convolutional neural network with the single-frame taillight illumination status annotation dataset and use the deep convolutional neural network to perform feature extraction on the plurality of image frames.

12. The system of claim 11 wherein the feature extraction is performed using an object detection process and application of bounding boxes to define the detection of objects in the image frame.

13. The system of claim 11 being further configured to perform local feature extraction on each side of a taillight mask of the single-frame taillight mask dataset.

14. The system of claim 10 being further configured to perform feature aggregation.

15. The system of claim 14 wherein the feature aggregation is further configured to integrate an extracted feature with its feature counterpart in a previous image frame within a predefined time window.

16. The system of claim 10 wherein the taillight illumination status includes an indication from the group consisting of:

the taillight is invisible or occluded, the taillight is visible but not illuminated or dark, the taillight is visible and illuminated or bright, and the taillight status is unknown.

17. The system of claim 10 wherein the taillight illumination status includes an inferred indication from the group consisting of:

brake, right turn on, left turn on, and hazard on.

18. The system of claim 10 being further configured to perform an autonomous vehicle control signal or route planning modification based on the recognized taillight illumination status of the proximate vehicle.

19. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive a plurality of image frames from one or more image-generating devices of an autonomous vehicle;

use a single-frame taillight illumination status annotation dataset and a single-frame taillight mask dataset to recognize a taillight illumination status of a proximate vehicle identified in an image frame of the plurality of image frames, the single-frame taillight illumination status annotation dataset including one or more taillight illumination status conditions of a right or left vehicle taillight signal, the single-frame taillight mask dataset including annotations to isolate a taillight region of a vehicle; and

use a multi-frame taillight illumination status dataset to recognize a taillight illumination status of the proximate vehicle in multiple image frames of the plurality of image frames, the multiple image frames being in temporal succession.

20. The non-transitory machine-useable storage medium of claim 19 being further configured to train a deep convolutional neural network with the single-frame taillight illumination status annotation dataset and use the deep convolutional neural network to perform feature extraction on the plurality of image frames.