System and method for detecting taillight signals of a vehicle

• US 10,387,736 B2
• Filed: 09/20/2017
• Issued: 08/20/2019
• Est. Priority Date: 09/20/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Wang, Yijie, Zhu, Ligeng, Wang, Panqu, Chen, Pengfei
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US15/709,832
• Publication Number:
  US 20190087672A1
• Time in Patent Office: 699 Days
• CPC Class Codes: B60Y 2200/11 Passenger cars; Automobiles
• CPC Class Codes: B60Y 2200/12 Motorcycles, Trikes; Quads; Scooters
• CPC Class Codes: B60Y 2200/143 Busses
• CPC Class Codes: B60Y 2200/22 Agricultural vehicles
• CPC Class Codes: B60Y 2200/252 Snowmobiles
• CPC Class Codes: B60Y 2200/30 Railway vehicles
• CPC Class Codes: B60Y 2200/51 Aeroplanes
• CPC Class Codes: B60Y 2200/52 Helicopters
• CPC Class Codes: B60Y 2200/86 Carts; Golf carts
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 2201/07 Target detection

Abstract

A system method for detecting taillight signals of a vehicle using a convolutional neural network is disclosed. A particular embodiment includes: receiving a plurality of images from one or more image-generating devices; generating a frame for each of the plurality of images; generating a ground truth, wherein the ground truth includes a labeled image with one of the following taillight status conditions for a right or left taillight signal of the vehicle: (1) an invisible right or left taillight signal, (2) a visible but not illuminated right or left taillight signal, and (3) a visible and illuminated right or left taillight signal; creating a first dataset including the labeled images corresponding to the plurality of images, the labeled images including one or more of the taillight status conditions of the right or left taillight signal; and creating a second dataset including at least one pair of portions of the plurality of images, wherein the at least one pair of portions of the plurality of the images are in temporal succession.

Claims

1. A method of detecting a taillight signal of a vehicle, the method comprising:

receiving a plurality of images from one or more image-generating devices;

generating a frame for each of the plurality of images;

generating a ground truth, wherein the ground truth includes a labeled image with one of the following taillight status conditions for a right or left taillight signal of the vehicle;

(1) an invisible right or left taillight signal, (2) a visible but not illuminated right or left taillight signal, and (3) a visible and illuminated right or left taillight signal;

creating a first dataset including the labeled images corresponding to the plurality of images, the labeled images including one or more of the taillight status conditions of the right or left taillight signal; and

creating a second dataset including at least one pair of portions of the plurality of images, wherein the at least one pair of portions of the plurality of the images are in temporal succession.

2. The method of the claim 1 further including generating a loss function by combining the first and the second dataset.

3. The method of the claim 1 further including determining a confidence level using parameters.

4. The method of the claim 1 further including outputting parameters to a temporal fusion process.

5. The method of the claim 1 further including adjusting parameters of a temporal fusion process.

6. The method of the claim 1 wherein generating the ground truth further includes labelling the plurality of data by adding annotations manually.

7. The method of the claim 1 further including determining a length of time of a cycle of taillight signal on and taillight signal off.

8. The method of the claim 1 wherein the taillight signal includes a turn signal, a brake signal, and an emergency stop signal.

9. The method of the claim 1 further including providing statistics of temporal patterns of taillight signals to a temporal fusion process.

10. A system comprising:

a data processor; and

a taillight signal recognition processing module, executable by the data processor, the taillight signal recognition processing module being configured to perform a taillight signal recognition operation using a convolutional neural network, the taillight signal recognition operation being configured to;

receive a plurality of images from one or more image-generating devices;

generate a frame for each of the plurality of images;

generate a ground truth, wherein the ground truth includes a labeled image with one of the following taillight status conditions for a right or left taillight signal of the vehicle;

(1) an invisible right or left taillight signal, (2) a visible but not illuminated right or left taillight signal, and (3) a visible and illuminated right or left taillight signal;

create a first dataset including the labeled images corresponding to the plurality of images, the labeled images including one or more of the taillight status conditions of the right or left taillight signal; and

create a second dataset including at least one pair of portions of the plurality of images, wherein the at least one pair of portions of the plurality of the images are in temporal succession.

11. The system of the claim 10 being further configured to generate a loss function by combining the first and the second dataset.

12. The system of the claim 10 being further configured to determine a confidence level using parameters.

13. The system of the claim 10 being further configured to output parameters to a temporal fusion process.

14. The system of the claim 10 being further configured to adjust parameters of a temporal fusion process.

15. The system of the claim 10 being further configured to generate the ground truth by labelling the plurality of data by adding annotations manually.

16. The system of the claim 10 being further configured to determine a length of time of a cycle of taillight signal on and taillight signal off.

17. The system of the claim 10 wherein the taillight signal includes a turn signal, a brake signal, and an emergency stop signal.

18. The system of the claim 10 being further configured to provide statistics of temporal patterns of taillight signals to a temporal fusion process.