Method and system for lane detection

• US 10,867,190 B1
• Filed: 11/27/2019
• Issued: 12/15/2020
• Est. Priority Date: 11/27/2019
• Status: Active Grant

• Current Assignee: Almotive KFT
• Original Assignee: AImotive Kft
• Inventors: Vajna, Szabolcs, Jozsa, Csaba Mate, Szolgay, Daniel, Zsamboki, Richard, Kozma, Daniel Akos
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US16/697,766
• Time in Patent Office: 384 Days
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Methods, systems, and computer program products for lane detection. An image processing module is trained by machine learning, and used to generate correspondence mapping data based on an image pair of a first and second images. The correspondence mapping data defines correspondence between a first lane boundary group of the first image and a second lane boundary group of the second image. An image space data block of image space detection pairs is then generated based on the correspondence mapping data, and a three-dimensional lane detection data block generated using triangulation based on first and second parts of the image space data block corresponding to respective first and second members of the image space detection pairs.

Claims

1. A method for lane detection comprising:

generating, by an image processing module trained by machine learning, output data based on an image pair including a first image having a first lane boundary group and a second image having a second lane boundary group, the output data including correspondence mapping data that defines a correspondence between the first lane boundary group and the second lane boundary group,

generating an image space detection data block of image space detection pairs based on the output data, and

generating a 3D lane detection data block using triangulation and calibration data corresponding to the first image and the second image based on a first part of the image space detection data block corresponding to a first member of the image space detection pairs and a second part of the image space detection data block corresponding to a second member of the image space detection pairs.

2. The method of claim 1, wherein:

the correspondence mapping data is comprised by a correspondence mapping data block, and

the output data further comprises;

a first image raw detection data block that defines an arrangement of each first member of the first lane boundary group, and

a second image raw detection data block that defines an arrangement of each second member of the second lane boundary group.

3. The method of claim 2, wherein:

the output data is generated for a coordinate grid represented by a coordinate grid tensor having a plurality of grid tensor elements,

the first image raw detection data block is a first image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of first tensor elements each comprising a respective first distance value measured from a corresponding grid tensor element of a closest first lane boundary point of the first lane boundary group in a first search direction in the first image,

the second image raw detection data block is a second image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of second tensor elements each comprising a respective second distance value measured from a corresponding grid tensor element of a closest second lane boundary point of the second lane boundary group in a second search direction in the second image, and

the correspondence mapping data block is a correspondence mapping tensor corresponding to the coordinate grid tensor and having a plurality of third tensor elements characterizing whether the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary, and

the image space detection data block is generated by using the coordinate grid tensor to generate a tensor of the image space detection pairs.

4. The method of claim 3, wherein:

the correspondence mapping tensor is a pairing correspondence mapping tensor, each of its third tensor elements having a pairing probability value characterizing a probability that the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary and are positioned within a window size from a coordinate grid tensor element corresponding to the respective third tensor element, and

the third tensor elements having a respective probability value above a predetermined first threshold are selected as foreground third tensor elements,

wherein the foreground third tensor elements and the corresponding first and second tensor elements are used as the output data from which the image space detection data block is generated.

5. The method of claim 4, wherein the first search direction and the second search direction are selected as one of the following combinations of respective search directions:

a left direction on the first image and the left direction on the second image, the left direction on the first image and a right direction on the second image, the right direction on the first image and a left direction on the second image, and the right direction on the first image and the right direction on the second image.

6. The method of claim 4, wherein:

the correspondence mapping tensor comprises at least two lane boundary-type selective correspondence mapping tensors, one for each of at least two different lane types,

the pairing probability value is a lane boundary-type selective probability value further characterizing the probability that a lane boundary corresponding to the closest first lane boundary point and the closest second lane boundary point is of a certain lane boundary-type, and

respective 3D lane detection data blocks are generated based on the at least two lane boundary-type selective correspondence mapping tensors for the at least two different lane types.

7. The method of claim 2, wherein:

the arrangement of each first member having a first index of the first lane boundary group in the first image is defined by first model parameters in the first image raw detection data block,

the arrangement of each second member having a second index of the second lane boundary group in the second image is defined by second model parameters in the second image raw detection data block, and

the correspondence mapping data block determines correspondences between a respective first index and a respective second index based on whether the first index and the second index correspond to the same lane boundary.

8. The method of claim 1, wherein the correspondence mapping data further defines the arrangement of each first member of the first lane boundary group and the arrangement of each second member of the second lane boundary group.

9. The method of claim 1, wherein:

the correspondence mapping data comprises first channel correspondence mapping data elements corresponding to the first image and second channel correspondence mapping data elements corresponding to the second image,

each of the first channel correspondence mapping data elements and the second channel correspondence mapping data elements defines a label characterizing whether the respective data element corresponds to a lane boundary, wherein the same label corresponds to the same lane boundary and different labels correspond to different lane boundaries, and

the data elements of the first channel correspondence mapping data elements and the second channel correspondence mapping data elements corresponding to the lane boundary are selected as foreground correspondence mapping data elements that are used as the output data from which the image space detection data block is generated.

10. The method of claim 1, wherein the image processing module is implemented by a neural network.

11. The method of claim 1, wherein the output data is generated by the image processing module based on, in addition to actual frames of the first image and the second image, at least one additional frame of the first image and the second image preceding or succeeding the actual frames.

12. The method of claim 1, wherein the image pair is a stereo image pair of a left image and a right image.

13. A system for lane detection comprising:

one or more processors; and

a memory coupled to the one or more processors and including program code that, when executed by at least one of the one or more processors, causes the system to;

generate, by an image processing module trained by machine learning, output data based on an image pair of a first image having a first lane boundary group and a second image having a second lane boundary group, the output data including correspondence mapping data that defines a correspondence between the first lane boundary group and the second lane boundary group,

generate an image space detection data block of image space detection pairs based on the output data, and

generate a 3D lane detection data block using triangulation and calibration data corresponding to the first image and the second image based on a first part of the image space detection data block corresponding to a first member of the image space detection pairs and a second part of the image space detection data block corresponding to a second member of the image space detection pairs.

14. The system of claim 13, wherein:

the correspondence mapping data is comprised by a correspondence mapping data block, and

the output data further comprises;

a first image raw detection data block that defines an arrangement of each first member of the first lane boundary group, and

a second image raw detection data block that defines an arrangement of each second member of the second lane boundary group.

15. The system of claim 14, wherein:

the output data is generated for a coordinate grid represented by a coordinate grid tensor having a plurality of grid tensor elements,

the first image raw detection data block is a first image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of first tensor elements each comprising a respective first distance value measured from a corresponding grid tensor element of a closest first lane boundary point of the first lane boundary group in a first search direction in the first image,

the second image raw detection data block is a second image raw detection tensor corresponding to the coordinate grid tensor and having a plurality of second tensor elements each comprising a respective second distance value measured from a corresponding grid tensor element of a closest second lane boundary point of the second lane boundary group in a second search direction in the second image, and

the correspondence mapping data block is a correspondence mapping tensor corresponding to the coordinate grid tensor and having a plurality of third tensor elements characterizing whether the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary, and

the image space detection data block is generated by using the coordinate grid tensor to generate a tensor of the image space detection pairs.

16. The system of claim 15, wherein:

the correspondence mapping tensor is a pairing correspondence mapping tensor, each of its third tensor elements having a pairing probability value characterizing a probability that the closest first lane boundary point and the closest second lane boundary point correspond to the same lane boundary and are positioned within a window size from a coordinate grid tensor element corresponding to the respective third tensor element, and

the third tensor elements having a respective probability values above a predetermined first threshold are selected as foreground third tensor elements, wherein the foreground third tensor elements and the corresponding first tensor elements and second tensor elements are used as the output data from which the image space detection data block is generated.

17. The system of claim 16, wherein the first search direction and the second search direction are selected as one of the following combinations of respective search directions:

a left direction on the first image and the left direction on the second image, the left direction on the first image and a right direction on the second image, the right direction on the first image and a left direction on the second image, and the right direction on the first image and the right direction on the second image.

18. The system of claim 16, wherein:

the correspondence mapping tensor comprises at least two lane boundary-type selective correspondence mapping tensors, one for each of at least two different lane types,

the pairing probability value is a lane boundary-type selective probability value further characterizing the probability that a lane boundary corresponding to the closest first lane boundary point and the closest second lane boundary point is of a certain lane boundary-type, and

respective 3D lane detection data blocks are generated based on the at least two lane boundary-type selective correspondence mapping tensors for the at least two different lane types.

19. The system of claim 14, wherein:

the arrangement of each first member having a first index of the first lane boundary group in the first image is defined by first model parameters in the first image raw detection data block,

the arrangement of each second member having a second index of the second lane boundary group in the second image is defined by second model parameters in the second image raw detection data block, and

the correspondence mapping data block determines correspondences between a respective first index and a respective second index based on whether the first index and the second index correspond to the same lane boundary.

20. The system of claim 13, wherein the correspondence mapping data further defines the arrangement of each first member of the first lane boundary group and the arrangement of each second member of the second lane boundary group.

21. The system of claim 13, wherein:

the correspondence mapping data comprises first channel correspondence mapping data elements corresponding to the first image and second channel correspondence mapping data elements corresponding to the second image,

each of the first channel correspondence mapping data elements and second channel correspondence mapping data elements defines a label characterizing whether the respective data element corresponds to a lane boundary, wherein the same label corresponds to the same lane boundary and different labels correspond to different lane boundaries, and

the data elements of the first channel correspondence mapping data elements and the second channel correspondence mapping data elements corresponding to the lane boundary are selected as foreground correspondence mapping data elements that are used as the output data from which the image space detection data block of image space detection pairs is generated.

22. The system of claim 13, wherein the image processing module is implemented by a neural network.

23. The system of claim 13, wherein the output data is generated by the image processing module based on, in addition to actual frames of the first image and the second image, at least one additional frame of the first image and the second image preceding or succeeding the actual frames.

24. The system of claim 13, wherein the image pair is a stereo image pair of a left image and a right image.

25. A method for training an image processing module, comprising:

controlling a 3D projection loss of the 3D lane detection data block during training of the image processing module by modifying learnable parameters of the image processing module so that the image processing module is configured to generate output data based on an image pair of a first image having a first lane boundary group and a second image having a second lane boundary group, the output data including correspondence mapping data that defines a correspondence between the first lane boundary group and the second lane boundary group, wherein;

an image space detection data block is generated based on the output data, and

the 3D lane detection data block is generated using triangulation and calibration data corresponding to the first image and the second image based on a first part of the image space detection data block corresponding to a first member of a plurality of image space detection pairs and a second part of the image space detection data block corresponding to a second member of the plurality of image space detection pairs.

26. A system for training an image processing module, the system comprising:

a 3D projection loss module adapted for controlling the 3D lane detection data block during the training of the image processing module by modifying learnable parameters of the image processing module so that the image processing module is configured to generate output data based on an image pair of a first image having a first lane boundary group and a second image having a second lane boundary group, the output data including correspondence mapping data that defines a correspondence between the first lane boundary group and the second lane boundary group, wherein

an image space detection data block is generated based on the output data, and

the 3D lane detection data block is generated using triangulation and calibration data corresponding to the first image and the second image based on a first part of the image space detection data block corresponding to a first member of a plurality of image space detection pairs and a second part of the image space detection data block corresponding to a second member of the plurality of image space detection pairs.