Systems and methods for non-obstacle area detection

• US 9,905,015 B2
• Filed: 08/24/2017
• Issued: 02/27/2018
• Est. Priority Date: 09/18/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Dane, Gokce, Bhaskaran, Vasudev
• Primary Examiner(s): ALLISON, ANDRAE S
• Application Number:
  US15/685,390
• Publication Number:
  US 20180012367A1
• Time in Patent Office: 187 Days
• Field of Search: None
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/507 from shading G06T7/586 takes precedence
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A method performed by an electronic device is described. The method includes generating a depth map of a scene external to a vehicle. The method also includes performing first processing in a first direction of a depth map to determine a first non-obstacle estimation of the scene. The method also includes performing second processing in a second direction of the depth map to determine a second non-obstacle estimation of the scene. The method further includes combining the first non-obstacle estimation and the second non-obstacle estimation to determine a non-obstacle map of the scene. The combining includes combining comprises selectively using a first reliability map of the first processing and/or a second reliability map of the second processing The method additionally includes navigating the vehicle using the non-obstacle map.

Claims

1. A method performed by an electronic device, comprising:

generating a depth map of a scene external to a vehicle;

performing first processing in a first direction of the depth map to determine a first non-obstacle estimation of the scene;

performing second processing in a second direction of the depth map to determine a second non-obstacle estimation of the scene;

combining the first non-obstacle estimation and the second non-obstacle estimation to determine a non-obstacle map of the scene, wherein the combining comprises selectively using at least one element selected from the group consisting of a first reliability map of the first processing and a second reliability map of the second processing; and

navigating the vehicle using the non-obstacle map.

2. The method of claim 1, wherein navigating the vehicle comprises providing at least the non-obstacle map as input to an advanced driver assistance systems (ADAS) to regulate at least one of speed or steering of the vehicle.

3. The method of claim 1, wherein the navigating comprises using the non-obstacle map to identify a region of interest that is used by at least one of an object detection algorithm or a lane detection algorithm.

4. The method of claim 1, wherein the vehicle is an autonomous automobile or aerial drone, wherein the navigating comprises navigating the autonomous automobile or aerial drone without human input using the non-obstacle map.

5. The method of claim 1, further comprising generating the depth map using a stereo image pair that is captured by two cameras attached to the vehicle.

6. The method of claim 1, further comprising generating the depth map using a sequence of video images that are captured by a camera attached to the vehicle.

7. The method of claim 1, wherein combining the first non-obstacle estimation and the second non-obstacle estimation further comprises:

performing both the first processing and the second processing in parallel; and

merging the first non-obstacle estimation and the second non-obstacle estimation based on the first reliability map and the second reliability map, wherein a given pixel is identified as a non-obstacle area in the non-obstacle map by comparing a reliability value for the given pixel in the first reliability map and the second reliability map.

8. The method of claim 1, wherein combining the first non-obstacle estimation and the second non-obstacle estimation comprises:

performing first processing of the depth map;

obtaining the first reliability map that includes a reliability value for a plurality of pixels in the first reliability map;

determining reliable regions that include pixels of the first reliability map that have reliability values greater than a threshold;

determining unreliable regions that include pixels of the first reliability map that do not have reliability values greater than the threshold;

identifying the reliable regions of the first reliability map as non-obstacle areas; and

performing second processing on the unreliable regions of the first reliability map to determine whether the unreliable regions are non-obstacle areas.

9. The method of claim 1, further comprising determining the first reliability map, comprising:

determining a segment fitting error for a given segment of pixels in the depth map based on a difference between estimated linear model parameters for the given segment and pre-determined linear model parameters, wherein the pre-determined linear model parameters are selected from among a plurality of road condition models, wherein each of the road condition models has a corresponding set of linear model parameters;

determining a reliability value for the given segment by comparing the segment fitting error to a first estimation threshold; and

applying the reliability value for the given segment to at least one pixel in the given segment.

10. The method of claim 1, further comprising determining the second reliability map, comprising:

determining whether the depth value of a given pixel in the depth map is within a range of a mode of the depth histogram, wherein the given pixel has a high reliability value when the depth value of the given pixel is within the range of the mode of the depth histogram.

11. An electronic device, comprising:

at least one processor;

a memory in communication with the at least one processor; and

instructions stored in the memory, the instructions executable by the at least one processor to;

generate a depth map of a scene external to a vehicle;

perform first processing in a first direction of the depth map to determine a first non-obstacle estimation of the scene;

perform second processing in a second direction of the depth map to determine a second non-obstacle estimation of the scene;

combine the first non-obstacle estimation and the second non-obstacle estimation to determine a non-obstacle map of the scene, wherein the combining comprises selectively using at least one element selected from the group consisting of a first reliability map of the first processing and a second reliability map of the second processing; and

navigate the vehicle using the non-obstacle map.

12. The electronic device of claim 11, wherein the instructions executable by the at least one processor to navigate the vehicle comprise instructions executable by the at least one processor to provide at least the non-obstacle map as input to an advanced driver assistance systems (ADAS) to regulate at least one of speed or steering of the vehicle.

13. The electronic device of claim 11, wherein the instructions executable by the at least one processor to navigate the vehicle comprise instructions executable by the at least one processor to use the non-obstacle map to identify a region of interest that is used by at least one of an object detection algorithm or a lane detection algorithm.

14. The electronic device of claim 11, wherein the vehicle is an autonomous automobile or aerial drone, wherein the instructions executable by the at least one processor to navigate the vehicle comprise instructions executable by the at least one processor to navigate the autonomous automobile or aerial drone without human input using the non-obstacle map.

15. The electronic device of claim 11, further comprising a first camera and a second camera, wherein the first and second cameras are attached to the vehicle and in communication with the at least one processor, and wherein the instructions executable by the at least one processor to generate the depth map comprise instructions executable by the at least one processor to generate the depth map using a stereo image pair captured by the first and second cameras.

16. The electronic device of claim 11, further comprising at least one camera attached to the vehicle and in communication with the at least one processor, and wherein the instructions executable by the at least one processor to generate the depth map comprise instructions executable by the at least one processor to generate the depth map using a sequence of video images captured by the at least one camera.

17. The electronic device of claim 11, wherein the instructions executable by the at least one processor to combine the first non-obstacle estimation and the second non-obstacle estimation further comprises instructions executable by the at least one processor to:

perform both the first processing and the second processing in parallel; and

merge the first non-obstacle estimation and the second non-obstacle estimation based on the first reliability map and the second reliability map, wherein a given pixel is identified as a non-obstacle area in the non-obstacle map by comparing a reliability value for the given pixel in the first reliability map and the second reliability map.

18. The electronic device of claim 11, wherein the instructions executable by the at least one processor to combine the first non-obstacle estimation and the second non-obstacle estimation comprises instructions executable by the at least one processor to:

perform first processing of the depth map;

obtain the first reliability map that includes a reliability value for a plurality of pixels in the first reliability map;

determine reliable regions that include pixels of the first reliability map that have reliability values greater than a threshold;

determine unreliable regions that include pixels of the first reliability map that do not have reliability values greater than the threshold;

identify the reliable regions of the first reliability map as non-obstacle areas; and

perform second processing on unreliable regions of the first reliability map to determine whether the unreliable regions are non-obstacle areas.

19. The electronic device of claim 11, further comprising instructions executable by the at least one processor to determine the first reliability map, comprising instructions executable by the at least one processor to:

determine a segment fitting error for a given segment of pixels in the depth map based on a difference between estimated linear model parameters for the given segment and pre-determined linear model parameters, wherein the pre-determined linear model parameters are selected from among a plurality of road condition models, wherein each of the road condition models has a corresponding set of linear model parameters;

determine a reliability value for the given segment by comparing the segment fitting error to a first estimation threshold; and

apply the reliability value for the given segment to at least one pixel in the given segment.

20. The electronic device of claim 11, further comprising instructions executable by the at least one processor to determine the second reliability map, comprising instructions executable by the at least one processor to:

determine whether the depth value of a given pixel in the depth map is within a range of a mode of the depth histogram, wherein the given pixel has a high reliability value when the depth value of the given pixel is within the range of the mode of the depth histogram.

21. A computer-program product, comprising a non-transitory tangible computer-readable medium having instructions thereon, the instructions comprising:

code for causing an electronic device to generate a depth map of a scene external to a vehicle;

code for causing the electronic device to perform first processing in a first direction of the depth map to determine a first non-obstacle estimation of the scene;

code for causing the electronic device to perform second processing in a second direction of the depth map to determine a second non-obstacle estimation of the scene;

code for causing the electronic device to combine the first non-obstacle estimation and the second non-obstacle estimation to determine a non-obstacle map of the scene, wherein the combining comprises selectively using at least one element selected from the group consisting of a first reliability map of the first processing and a second reliability map of the second processing; and

code for causing the electronic device to navigate the vehicle using the non-obstacle map.

22. The computer-program product of claim 21, wherein the code for causing the electronic device to navigate the vehicle comprises code for causing the electronic device to provide at least the non-obstacle map as input to an advanced driver assistance systems (ADAS) to regulate at least one of speed or steering of the vehicle.

23. The computer-program product of claim 21, wherein the code for causing the electronic device to navigate comprises code for causing the electronic device to use the non-obstacle map to identify a region of interest that is used by at least one of an object detection algorithm or a lane detection algorithm.

24. The computer-program product of claim 21, wherein the vehicle is an autonomous automobile or aerial drone, wherein the code for causing the electronic device to navigate comprises code for causing the electronic device to navigate the autonomous automobile or aerial drone without human input using the non-obstacle map.

25. The computer-program product of claim 21, further comprising code for causing the electronic device to generate the depth map using a stereo image pair that is captured by two cameras attached to the vehicle.

26. The computer-program product of claim 21, further comprising code for causing the electronic device to generate the depth map using a sequence of video images that are captured by a camera attached to the vehicle.

27. The computer-program product of claim 21, wherein the code for causing the electronic device to combine the first non-obstacle estimation and the second non-obstacle estimation further comprises code for causing the electronic device to:

perform both the first processing and the second processing in parallel; and

merge the first non-obstacle estimation and the second non-obstacle estimation based on the first reliability map and the second reliability map, wherein a given pixel is identified as a non-obstacle area in the non-obstacle map by comparing a reliability value for the given pixel in the first reliability map and the second reliability map.

28. The computer-program product of claim 21, wherein the code for causing the electronic device to combine the first non-obstacle estimation and the second non-obstacle estimation comprises code for causing the electronic device to:

perform first processing of the depth map;

obtain the first reliability map that includes a reliability value for a plurality of pixels in the first reliability map;

determine reliable regions that include pixels of the first reliability map that have reliability values greater than a threshold;

determine unreliable regions that include pixels of the first reliability map that do not have reliability values greater than the threshold;

identify the reliable regions of the first reliability map as non-obstacle areas; and

perform second processing on unreliable regions of the first reliability map to determine whether the unreliable regions are non-obstacle areas.

29. The computer-program product of claim 21, further comprising code for causing the electronic device to determining the first reliability map, comprising code for causing the electronic device to:

determine a segment fitting error for a given segment of pixels in the depth map based on a difference between estimated linear model parameters for the given segment and pre-determined linear model parameters, wherein the pre-determined linear model parameters are selected from among a plurality of road condition models, wherein each of the road condition models has a corresponding set of linear model parameters;

determine a reliability value for the given segment by comparing the segment fitting error to a first estimation threshold; and

apply the reliability value for the given segment to at least one pixel in the given segment.

30. The computer-program product of claim 21, further comprising code for causing the electronic device to determine the second reliability map, comprising code for causing the electronic device to:

determine whether the depth value of a given pixel in the depth map is within a range of a mode of the depth histogram, wherein the given pixel has a high reliability value when the depth value of the given pixel is within the range of the mode of the depth histogram.