Systems and methods for depth map sampling

• US 10,282,591 B2
• Filed: 08/24/2015
• Issued: 05/07/2019
• Est. Priority Date: 08/24/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lindner, Albrecht Johannes, Slobodyanyuk, Volodimir, Verrall, Stephen Michael, Atanassov, Kalin Mitkov
• Primary Examiner(s): Anderson, II, James M
• Application Number:
  US14/833,573
• Publication Number:
  US 20170061632A1
• Time in Patent Office: 1,352 Days
• Field of Search: None
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

An electronic device is described. The electronic device includes a camera configured to capture an image of a scene. The electronic device also includes an image segmentation mapper configured to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene. The electronic device further includes a memory configured to store the image and the spatial coordinates. The electronic device additionally includes a LIDAR (light+radar) unit, the LIDAR unit steerable to selectively obtain depth values corresponding to at least a subset of the spatial coordinates. The electronic device further includes a depth mapper configured to generate a depth map of the scene based on the depth values and the spatial coordinates.

Claims

1. An electronic device, comprising:

a camera configured to capture an image of a scene;

an image segmentation mapper configured to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;

a memory configured to store the image and the spatial coordinates;

a LIDAR (light+radar) unit, the LIDAR unit steerable to selectively obtain depth values of the scene corresponding to the spatial coordinates; and

a depth mapper configured to generate a depth map of the scene based on the depth values and the spatial coordinates.

2. The electronic device of claim 1, wherein the image segmentation mapper is configured to perform segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

3. The electronic device of claim 1, wherein the LIDAR unit is configured to perform a coarse scan over a region of the scene containing a substantially uniform object within the scene.

4. The electronic device of claim 1, wherein the sampling point for each image segment corresponds to a centroid of the segment.

5. The electronic device of claim 1, wherein the image segmentation mapper is configured to provide the sampling point for each image segment to the LIDAR unit and to provide the segments of the image to the depth mapper.

6. The electronic device of claim 1, wherein the depth mapper is configured to generate the depth map by merging the segments with corresponding depth values.

7. The electronic device of claim 1, wherein the depth mapper is configured to generate the depth map by populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

8. The electronic device of claim 1, wherein the number of depth values obtained by the LIDAR unit is configured to be adjusted based on feedback from a prior depth map.

9. A method, comprising:

capturing an image of a scene;

performing segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;

obtaining, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

generating a depth map of the scene based on the depth values and the spatial coordinates.

10. The method of claim 9, wherein performing segmentation of the image comprises performing segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

11. The method of claim 9, wherein the LIDAR unit is configured to perform a coarse scan over a region of the scene containing a substantially uniform object within the scene.

12. The method of claim 9, wherein generating the depth map comprises merging the segments with corresponding depth values.

13. The method of claim 9, wherein generating the depth map comprises populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

14. The method of claim 9, further comprising adjusting the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

15. An apparatus, comprising:

means for capturing an image of a scene;

means for performing segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;

means for obtaining, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

means for generating a depth map of the scene based on the depth values and the spatial coordinates.

16. The apparatus of claim 15, wherein the means for performing segmentation of the image comprise means for performing segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

17. The apparatus of claim 15, wherein the sampling point for each image segment corresponds to a centroid of the segment.

18. The apparatus of claim 15, wherein the means for generating the depth map comprise means for merging the segments with corresponding depth values.

19. The apparatus of claim 15, wherein the means for generating the depth map comprise means for populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

20. The apparatus of claim 15, further comprising means for adjusting the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

21. A computer-program product, comprising a non-transitory tangible computer-readable medium having instructions thereon, the instructions comprising:

code for causing an electronic device to capture an image of a scene;

code for causing the electronic device to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;

code for causing the electronic device to obtain, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and

code for causing the electronic device to generate a depth map of the scene based on the depth values and the spatial coordinates.

22. The computer-program product of claim 21, wherein the code for causing the electronic device to perform segmentation of the image comprises code for causing the electronic device to perform segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

23. The computer-program product of claim 21, wherein the sampling point for each image segment corresponds to a centroid of the segment.

24. The computer-program product of claim 21, wherein the code for causing the electronic device to generate the depth map comprises code for causing the electronic device to merge the segments with corresponding depth values.

25. The computer-program product of claim 21, wherein the code for causing the electronic device to generate the depth map comprises code for causing the electronic device to populate each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

26. The computer-program product of claim 21, further comprising code for causing the electronic device to adjust the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.