SYSTEM AND METHOD FOR USING TRIPLET LOSS FOR PROPOSAL FREE INSTANCE-WISE SEMANTIC SEGMENTATION FOR LANE DETECTION

• US 20190065867A1
• Filed: 08/23/2017
• Published: 02/28/2019
• Est. Priority Date: 08/23/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: Tusimple
• Inventors: HUANG, Zehua, WANG, Panqu, CHEN, Pengfei, LI, Tian
• Granted Patent: US 10,303,956 B2
• Time in Patent Office: Days
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A system and method for using triplet loss for proposal free instance-wise semantic segmentation for lane detection are disclosed. A particular embodiment includes: receiving image data from an image generating device mounted on an autonomous vehicle; performing a semantic segmentation operation or other object detection on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis and producing corresponding semantic segmentation prediction data; performing a triplet loss calculation operation using the semantic segmentation prediction data to identify different instances of objects with similar object category labels found in the image data; and determining an appropriate vehicle control action for the autonomous vehicle based on the different instances of objects identified in the image data.

Claims

1. A system comprising:

a data processor; and

an image processing and lane detection module, executable by the data processor, the image processing and lane detection module being configured to perform an image processing and lane detection operation configured to;

receive image data from an image generating device mounted on an autonomous vehicle;

perform a semantic segmentation operation or other object detection on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis and produce corresponding semantic segmentation prediction data;

perform a triplet loss calculation operation using the semantic segmentation prediction data to identify different instances of objects with similar object category labels found in the image data; and

determine an appropriate vehicle control action for the autonomous vehicle based on the different instances of objects identified in the image data.

2. The system of claim 1 wherein the image generating device is one or more cameras.

3. The system of claim 1 wherein the image data corresponds to at least one frame from a video stream generated by one or more cameras.

4. The system of claim 1 being further configured to output lane detection data to a vehicle control subsystem of the autonomous vehicle.

5. The system of claim 1 wherein a neural network is used in the semantic segmentation operation to identify and label objects in the image data with object category labels on a per-pixel basis.

6. The system of claim 1 wherein a neural network is used in the triplet loss calculation operation to identify different instances of objects with similar object category labels found in the image data.

7. The system of claim 1 wherein the triplet loss calculation operation being configured to select an anchor pixel from the image data, select a second pixel proximally located relative to the anchor pixel, select a third pixel distally located relative to the anchor pixel, determine that the anchor pixel and the second pixel are associated with a same object instance, and determine that the anchor pixel and the third pixel are associated with a different object instance.

8. A method comprising:

receiving image data from an image generating device mounted on an autonomous vehicle;

performing a semantic segmentation operation or other object detection on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis and producing corresponding semantic segmentation prediction data;

performing a triplet loss calculation operation using the semantic segmentation prediction data to identify different instances of objects with similar object category labels found in the image data; and

determining an appropriate vehicle control action for the autonomous vehicle based on the different instances of objects identified in the image data.

9. The method of claim 8 wherein the image generating device is one or more cameras.

10. The method of claim 8 wherein the image data corresponds to at least one frame from a video stream generated by one or more cameras.

11. The method of claim 8 including outputting lane detection data to a vehicle control subsystem of the autonomous vehicle.

12. The method of claim 8 including using a neural network in the semantic segmentation operation to identify and label objects in the image data with object category labels on a per-pixel basis.

13. The method of claim 8 including using a neural network in the triplet loss calculation operation to identify different instances of objects with similar object category labels found in the image data.

14. The method of claim 8 wherein the triplet loss calculation operation including selecting an anchor pixel from the image data, selecting a second pixel proximally located relative to the anchor pixel, selecting a third pixel distally located relative to the anchor pixel, determining that the anchor pixel and the second pixel are associated with a same object instance, and determining that the anchor pixel and the third pixel are associated with a different object instance.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive image data from an image generating device mounted on an autonomous vehicle;

perform a semantic segmentation operation or other object detection on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis and produce corresponding semantic segmentation prediction data;

perform a triplet loss calculation operation using the semantic segmentation prediction data to identify different instances of objects with similar object category labels found in the image data; and

determine an appropriate vehicle control action for the autonomous vehicle based on the different instances of objects identified in the image data.

16. The non-transitory machine-useable storage medium of claim 15 wherein the image data corresponds to at least one frame from a video stream generated by one or more cameras.

17. The non-transitory machine-useable storage medium of claim 15 being further configured to output lane detection data to a vehicle control subsystem of the autonomous vehicle.

18. The non-transitory machine-useable storage medium of claim 15 wherein a neural network is used in the semantic segmentation operation to identify and label objects in the image data with object category labels on a per-pixel basis.

19. The non-transitory machine-useable storage medium of claim 15 wherein a neural network is used in the triplet loss calculation operation to identify different instances of objects with similar object category labels found in the image data.

20. The non-transitory machine-useable storage medium of claim 15 wherein the triplet loss calculation operation being configured to select an anchor pixel from the image data, select a second pixel proximally located relative to the anchor pixel, select a third pixel distally located relative to the anchor pixel, determine that the anchor pixel and the second pixel are associated with a same object instance, and determine that the anchor pixel and the third pixel are associated with a different object instance.