SYSTEMS AND METHODS FOR DEPTH MAP SAMPLING

• US 20170061632A1
• Filed: 08/24/2015
• Published: 03/02/2017
• Est. Priority Date: 08/24/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lindner, Albrecht Johannes, Slobodyanyuk, Volodimir, Verrall, Stephen Michael, Atanassov, Kalin Mitkov
• Granted Patent: US 10,282,591 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

An electronic device is described. The electronic device includes a camera configured to capture an image of a scene. The electronic device also includes an image segmentation mapper configured to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene. The electronic device further includes a memory configured to store the image and the spatial coordinates. The electronic device additionally includes a LIDAR (light+radar) unit, the LIDAR unit steerable to selectively obtain depth values corresponding to at least a subset of the spatial coordinates. The electronic device further includes a depth mapper configured to generate a depth map of the scene based on the depth values and the spatial coordinates.

Claims

1. An electronic device, comprising:

a camera configured to capture an image of a scene;

an image segmentation mapper configured to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene;

a memory configured to store the image and the spatial coordinates;

a LIDAR (light+radar) unit, the LIDAR unit steerable to selectively obtain depth values corresponding to at least a subset of the spatial coordinates; and

a depth mapper configured to generate a depth map of the scene based on the depth values and the spatial coordinates.

2. The electronic device of claim 1, wherein at least a portion of the image segments comprise non-uniform segments that define borders of an object within the image.

3. The electronic device of claim 1, wherein the image segmentation mapper is configured to perform segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

4. The electronic device of claim 1, wherein the LIDAR unit is configured to perform a coarse scan over a region of the scene containing a substantially uniform object within the scene.

5. The electronic device of claim 1, wherein the spatial coordinates of the segments correspond to centroids of the segments.

6. The electronic device of claim 1, wherein the image segmentation mapper is configured to provide the spatial coordinates to the LIDAR unit and to provide the segments of the image to the depth mapper.

7. The electronic device of claim 1, wherein the depth mapper is configured to generate the depth map by merging the segments with corresponding depth values.

8. The electronic device of claim 1, wherein the depth mapper is configured to generate the depth map by populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

9. The electronic device of claim 1, wherein the number of depth values obtained by the LIDAR unit is configured to be adjusted based on feedback from a prior depth map.

10. A method, comprising:

capturing an image of a scene;

performing segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene;

obtaining, by a LIDAR unit, depth values corresponding to at least a subset of the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

generating a depth map of the scene based on the depth values and the spatial coordinates.

11. The method of claim 10, wherein at least a portion of the image segments comprise non-uniform segments that define borders of an object within the image.

12. The method of claim 10, wherein performing segmentation of the image comprises performing segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

13. The method of claim 10, wherein the LIDAR unit is configured to perform a coarse scan over a region of the scene containing a substantially uniform object within the scene.

14. The method of claim 10, wherein generating the depth map comprises merging the segments with corresponding depth values.

15. The method of claim 10, wherein generating the depth map comprises populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

16. The method of claim 10, further comprising adjusting the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

17. An apparatus, comprising:

means for capturing an image of a scene;

means for performing segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene;

means for obtaining, by a LIDAR unit, depth values corresponding to at least a subset of the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

means for generating a depth map of the scene based on the depth values and the spatial coordinates.

18. The apparatus of claim 17, wherein at least a portion of the image segments comprise non-uniform segments that define borders of an object within the image.

19. The apparatus of claim 17, wherein the means for performing segmentation of the image comprise means for performing segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

20. The apparatus of claim 17, wherein the spatial coordinates of the segments correspond to centroids of the segments.

21. The apparatus of claim 17, wherein the means for generating the depth map comprise means for merging the segments with corresponding depth values.

22. The apparatus of claim 17, wherein the means for generating the depth map comprise means for populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

23. The apparatus of claim 17, further comprising means for adjusting the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

24. A computer-program product, comprising a non-transitory tangible computer-readable medium having instructions thereon, the instructions comprising:

code for causing an electronic device to capture an image of a scene;

code for causing the electronic device to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene;

code for causing the electronic device to obtain, by a LIDAR unit, depth values corresponding to at least a subset of the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

code for causing the electronic device to generate a depth map of the scene based on the depth values and the spatial coordinates.

25. The computer-program product of claim 24, wherein at least a portion of the image segments comprise non-uniform segments that define borders of an object within the image.

26. The computer-program product of claim 24, wherein the code for causing the electronic device to perform segmentation of the image comprises code for causing the electronic device to perform segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

27. The computer-program product of claim 24, wherein the spatial coordinates of the segments correspond to centroids of the segments.

28. The computer-program product of claim 24, wherein the code for causing the electronic device to generate the depth map comprises code for causing the electronic device to merge the segments with corresponding depth values.

29. The computer-program product of claim 24, wherein the code for causing the electronic device to generate the depth map comprises code for causing the electronic device to populate each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

30. The computer-program product of claim 24, further comprising code for causing the electronic device to adjust the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.