REAL-TIME 3D RECONSTRUCTION WITH POWER EFFICIENT DEPTH SENSOR USAGE

• US 20140139639A1
• Filed: 01/22/2014
• Published: 05/22/2014
• Est. Priority Date: 01/30/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wagner, Daniel, Mulloni, Alessandro
• Granted Patent: US 9,083,960 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

Embodiments disclosed facilitate resource utilization efficiencies in Mobile Stations (MS) during 3D reconstruction. In some embodiments, camera pose information for a first color image captured by a camera on an MS may be obtained and a determination may be made whether to extend or update a first 3-Dimensional (3D) model of an environment being modeled by the MS based, in part, on the first color image and associated camera pose information. The depth sensor, which provides depth information for images captured by the camera, may be disabled, when the first 3D model is not extended or updated.

Claims

1. A method on a Mobile Station (MS) comprising at least one camera and a depth sensor, the method comprising:

capturing a first image with at least one camera, the first image comprising color information for at least a portion of an environment being modeled by the MS;

obtaining camera pose information for the first image;

determining, based, in part, on the first captured image and the camera pose information for the first image, whether to extend a first 3-Dimensional (3D) model of the environment; and

disabling the depth sensor, when the first 3D model is not extended.

2. The method of claim 1, wherein disabling the depth sensor comprises at least one of:

turning off power supplied to the depth sensor, or

disabling functionality related to computation of a depth image.

3. The method of claim 1, further comprising:

enabling the depth sensor, when the first 3D model is extended.

4. The method of claim 3, wherein upon enabling the depth sensor, the first 3D model is extended by:

capturing a second image with the at least one camera;

augmenting the second image with depth information provided, at least in part, by the depth sensor; and

merging a partial model of the environment being modeled by the MS with the first 3D model of the environment, wherein the partial model is based, in part, on the second image, and the depth and camera pose information for the second image.

5. The method of claim 4, wherein information common to the first 3D model and the partial 3D model is removed from the partial 3D model prior to merging the partial 3D model with the first 3D model.

6. The method of claim 4, wherein the partial model is based on a downsampled lower resolution augmented second image.

7. The method of claim 4, wherein merging the partial model of the environment being modeled by the MS with the first 3D model of the environment, further comprises aligning the partial 3D model with the first 3D model based on the camera pose information for the second image.

8. The method of claim 7, wherein the alignment of the partial 3D model with the first 3D model is refined using Iterative Closest Point (ICP).

9. A Mobile Station (MS) comprising:

a camera, the camera to capture a first image comprising color information,

a depth sensor coupled to the camera; and

a processor coupled to the depth sensor and the camera, the processor configured to;

obtain camera pose information for the first image,

determine, based, in part, on the first image and the camera pose information for the first image, whether to extend a first 3-Dimensional (3D) model of an environment being modeled by the MS, and

disable the depth sensor, if the first 3D model is not extended.

10. The MS of claim 9, wherein to disable the depth sensor, the processor is configured to perform at least one of:

turn off power supplied to the depth sensor, or

disable functionality related to computation of a depth image.

11. The MS of claim 10, wherein the processor is further configured to:

enable the depth sensor, when the first 3D model is extended.

12. The MS of claim 11, wherein upon enabling the depth sensor:

the camera further captures a second image comprising color information;

the depth sensor further augments the second image captured by the camera with depth information; and

wherein the processor is further configured to;

obtain the augmented second image comprising color and depth information, and

merge a partial model of the environment being modeled by the MS with the first 3D model of the environment, wherein the partial model is based, in part, on the second image, and the depth and camera pose information for the second image.

13. The MS of claim 12, wherein prior to merging the partial 3D model with the first 3D model, the processor is further configured to:

remove information common to the first 3D model and the partial 3D model from the partial 3D model.

14. The MS of claim 12, wherein, the processor is configured to:

obtain the partial model based, in part, on a downsampled lower resolution version of the second image.

15. The MS of claim 12, wherein to merge the partial model of the environment being modeled by the MS with the first 3D model of the environment, the processor is configured to:

align the partial 3D model with the first 3D model based on the camera pose information associated with the second image.

16. The MS of claim 15, wherein the processor is configured to:

refine the alignment of the partial 3D model with the first 3D model using Iterative Closest Point (ICP).

17. An apparatus comprising:

imaging means, the imaging means to capture a first image comprising color information,

depth sensing means coupled to the imaging means, and

processing means coupled to the depth sensing means and the imaging means, the processing means further comprising;

means for obtaining imaging means pose information for the first image,

means for determining whether to extend a first 3-Dimensional (3D) model of an environment being modeled by the apparatus based, in part, on the first image and the imaging means pose information for the first image, and

means for disabling the depth sensing means, if the first 3D model is not extended.

18. The apparatus of claim 17, wherein means for disabling the depth sensing means further comprises at least one of:

means for turning off power supplied to the depth sensing means, or

means for disabling functionality related to computation of a depth image.

19. The apparatus of claim 17, wherein the processing means further comprises:

means for enabling the depth sensor, when the first 3D model is extended.

20. The apparatus of claim 19, wherein upon enabling the depth sensing means:

the imaging means further captures a second image comprising color information;

the depth sensing means further augments the second image with depth information; and

wherein the processing means further comprises;

means for obtaining the augmented second image comprising color and depth information, and

means for merging a partial model of the environment being modeled by the apparatus with the first 3D model of the environment, wherein the partial model is based, in part, on the augmented second image, and the imaging means pose information for the second image.

21. The apparatus of claim 20, wherein the processing means further comprises:

means for removing information common to the first 3D model and the partial 3D model from the partial 3D model prior to merging the partial 3D model with the first 3D model.

22. The apparatus of claim 20, wherein means for merging the partial model of the environment being modeled by the apparatus with the first 3D model of the environment, further comprises:

means for aligning the partial 3D model with the first 3D model based on the imaging means pose information associated with the second image.

23. The apparatus of claim 20, wherein the means for merging the partial model of the environment with the first 3D model of the environment, further comprises:

means for downsampling the augmented second image to a lower resolution augmented second image, and

wherein the means for merging operates on the downsampled lower resolution augmented second image.

24. A non-transitory computer-readable medium comprising instructions, which when executed by a processor, perform steps in a method on a Mobile Station (MS) comprising at least one camera and a depth sensor, the method comprising:

capturing a first image with at least one camera, the first image comprising color information for at least a portion of an environment being modeled by the MS;

obtaining camera pose information for the first image;

determining, based, in part, on the first captured image and the camera pose information for the first image, whether to extend a first 3-Dimensional (3D) model of the environment; and

disabling the depth sensor, when the first 3D model is not extended.

25. The computer-readable medium of claim 24, wherein disabling the depth sensor further comprises at least one of:

turning off power supplied to the depth sensor, or

disabling functionality related to computation of a depth image.

26. The computer-readable medium of claim 24, further comprising:

enabling the depth sensor, if the first 3D model is extended.

27. The computer-readable medium of claim 26, wherein upon enabling the depth sensor, the first 3D model is extended by:

augmenting the second image with depth information provided, at least in part, by the depth sensor; and

merging a partial model of the environment being modeled by the MS with the first 3D model of the environment, wherein the partial model is based, in part, on the second image, and the depth and camera pose information for the second image.

28. The computer-readable medium of claim 27, wherein information common to the first 3D model and the partial 3D model is either removed from or merged with the partial 3D model prior to merging the partial 3D model with the first 3D model.

29. The computer-readable medium of claim 27, wherein merging the partial model of the environment being modeled by the MS with the first 3D model of the environment, further comprises aligning the partial 3D model with the first 3D model based on the camera pose information for the second image.

30. The computer-readable medium of claim 27, wherein the partial 3D model is based on a downsampled lower resolution augmented second image.