REAL-TIME 3D RECONSTRUCTION WITH POWER EFFICIENT DEPTH SENSOR USAGE

• US 20150294499A1
• Filed: 06/03/2015
• Published: 10/15/2015
• Est. Priority Date: 01/30/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wagner, Daniel, Mulloni, Alessandro, Reitmayr, Gerhard, Knoblauch, Daniel
• Granted Patent: US 9,443,350 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/705 Pixels for depth measurement, e.g. RGBZ

Abstract

Embodiments disclosed facilitate resource utilization efficiencies in Mobile Stations (MS) during 3D reconstruction. In some embodiments, camera pose information for a first color image captured by a camera on an MS may be obtained and a determination may be made whether to extend or update a first 3-Dimensional (3D) model of an environment being modeled by the MS based, in part, on the first color image and associated camera pose information. The depth sensor, which provides depth information for images captured by the camera, may be disabled, when the first 3D model is not extended or updated.

Claims

1. A method on a Mobile Station (MS) comprising at least one camera and a depth sensor, the method comprising:

capturing a first image with the at least one camera, the first image comprising color information for an environment being modeled by the MS;

obtaining a camera pose for the first image;

determining, based, in part, on color information in the first image and the camera pose for the first image, whether to update a volumetric dataset associated with a first 3-Dimensional (3D) model of the environment; and

enabling the depth sensor to capture depth information when the first 3D model is updated, wherein the captured depth information is used, at least in part, to update the first 3D model using volumetric reconstruction.

2. The method of claim 1, wherein enabling the depth sensor comprises at least one of:

turning on power supplied to the depth sensor, or

enabling functionality related to computation of a depth image.

3. The method of claim 1, further comprising:

maintaining disablement of the depth sensor, when the first 3D model is not extended.

4. The method of claim 1, wherein a determination to update the volumetric dataset is made when:

the color information in the first image includes data pertaining to one or more points in the environment, which are unrepresented in the volumetric dataset;

or

the color information in the first image includes data pertaining to one or more points in the environment represented in the volumetric dataset and for which additional depth data is desired;

or

the first image is the last of a predetermined number of consecutive images with color information since the depth sensor was last enabled.

5. The method of claim 1, wherein upon enabling the depth sensor, the first 3D model is updated using volumetric reconstruction by:

capturing, with the at least one camera, a second image with color information;

augmenting the second image with the captured depth information provided by the depth sensor; and

fusing, based, in part, on a camera pose for the second image, the augmented second image with the volumetric dataset associated with the first 3D model.

6. The method of claim 5, wherein the volumetric dataset is represented using a Truncated Signed Distance Function (TDSF).

7. The method of claim 6, wherein fusing the augmented second image with the volumetric dataset comprises:

determining the camera pose for the second image;

projecting, based on the camera pose for the second image, one or more data samples in the TDSF into a depth map based on the augmented second image;

determining, for each of the one or more data samples projected into the depth map, a corresponding measured relative distance from a location of the projected data sample to a surface defined by the TDSF;

obtaining, for each of the one or more data samples projected into the depth map, a corresponding mapped TDSF value by mapping the corresponding measured relative distance to the TDSF; and

combining, for each of the one or more one or more data samples in the TDSF, the corresponding mapped TDSF value with a stored TDSF value for the data sample.

8. The method claim 7, wherein combining comprises:

applying a recursive filter to combine the corresponding mapped TDSF value with the stored TDSF value.

9. A Mobile Station (MS) comprising:

a camera, the camera to capture a first image comprising color information for an environment being modeled by the MS,

a depth sensor coupled to the camera; and

a processor coupled to the depth sensor and the camera, the processor configured to;

obtain a camera pose for the first image;

determine, based, in part, on color information in the first image and the camera pose for the first image, whether to update a volumetric dataset associated with a first 3-Dimensional (3D) model of the environment; and

enable the depth sensor to capture depth information when the first 3D model is updated, wherein the captured depth information is used, at least in part, to update the first 3D model using volumetric reconstruction.

10. The MS of claim 9, wherein to enable the depth sensor, the processor is configured to perform at least one of:

turn on power supplied to the depth sensor, or

enable functionality related to computation of a depth image.

11. The MS of claim 9, wherein the processor is further configured to:

maintain disablement of the depth sensor, when the first 3D model is not extended.

12. The MS of claim 9, wherein the processor is configured to make a determination to update the volumetric dataset when:

the color information in the first image includes data pertaining to one or more points in the environment, which are unrepresented in the volumetric dataset;

or

the color information in the first image includes data pertaining to one or more points in the environment represented in the volumetric dataset and for which additional depth data is desired;

or

the first image is the last of a predetermined number of consecutive images with color information since the depth sensor was last enabled.

13. The MS of claim 9, wherein to update first 3D model using volumetric reconstruction upon enabling the depth sensor, the processor is configured to:

capture, with the at least one camera, a second image with color information for the environment;

augment the second image with the captured depth information provided by the depth sensor; and

fuse, based, in part, on a camera pose for the second image, the augmented second image with the volumetric dataset associated with the first 3D model.

14. The MS of claim 13, wherein the volumetric dataset is represented using a Truncated Signed Distance Function (TDSF).

15. The MS of claim 14, wherein to fuse the augmented second image with the volumetric dataset, the processor is configured to:

determine the camera pose for the second image;

project, based on the camera pose of the second image, one or more data samples in the TDSF into a depth map based on the augmented second image;

determine, for each of the one or more data samples projected into the depth map, a corresponding measured relative distance from a location of the projected data sample to a surface defined by the TDSF;

obtain, for each of the one or more data samples projected into the depth map, a corresponding mapped TDSF value by mapping the corresponding measured relative distance to the TDSF; and

combine, for each of the one or more one or more data samples in the TDSF, the corresponding mapped TDSF value with a stored TDSF value for the data sample.

16. The MS of claim 15, wherein, the processor is configured to combine the corresponding mapped TDSF value with the stored TDSF value by applying a recursive filter.

17. An apparatus comprising:

imaging means, the imaging means to capture a first image comprising color information for an environment being modeled by the MS,

depth sensing means coupled to the imaging means, and

processing means coupled to the depth sensing means and the imaging means, the processing means further comprising;

means for obtaining an imaging means pose for the first image,

means for determining, based, in part, on color information in the first image and the imaging means pose for the first image, whether to update a volumetric dataset associated with a first 3-Dimensional (3D) model of the environment; and

means for enabling the depth sensing means to capture depth information when the first 3D model is updated, wherein the captured depth information is used, at least in part, by means for updating the first 3D model using volumetric reconstruction.

18. The apparatus of claim 17, wherein the means for enabling the depth sensing means comprises at least one of:

means for turning on power supplied to the depth sensing means, or

means for enabling functionality related to computation of a depth image.

19. The apparatus of claim 17, wherein the processing means further comprises:

means for maintaining disablement of the depth sensing means, when the first 3D model is not extended.

20. The apparatus of claim 17, wherein the means for determining determines to update the volumetric dataset when:

the color information in the first image includes data pertaining to one or more points in the environment, which are unrepresented in the volumetric dataset;

or

the color information in the first image includes data pertaining to one or more points in the environment represented in the volumetric dataset and for which additional depth data is desired;

or

the first image is the last of a predetermined number of consecutive images with color information since the depth sensing means was last enabled.

21. The apparatus of claim 17, wherein means for updating first 3D model using volumetric reconstruction upon enabling the depth sensor comprises:

means for capturing, with the at least one imaging means, a second image with color information for the environment;

means for augmenting the second image with the captured depth information provided by the depth sensing means; and

means for fusing, based, in part, on an imaging means pose for the second image, the augmented second image with the volumetric dataset associated with the first 3D model.

22. The apparatus of claim 21, wherein the volumetric dataset is represented using a Truncated Signed Distance Function (TDSF).

23. The apparatus of claim 22, wherein means for fusing the augmented second image with the volumetric dataset comprises:

means for determining the imaging means pose for the second image;

means for projecting, based on the imaging means pose for the second image, one or more data samples in the TDSF into a depth map based on the augmented second image;

means for determining, for each of the one or more data samples projected into the depth map, a corresponding measured relative distance from a location of the projected data sample to a surface defined by the TDSF;

means for obtaining, for each of the one or more data samples projected into the depth map, a corresponding mapped TDSF value by mapping the corresponding measured relative distance to the TDSF; and

means for combining, for each of the one or more one or more data samples in the TDSF, the corresponding mapped TDSF value with a stored TDSF value for the data sample.

24. A non-transitory computer-readable medium comprising instructions, which when executed by a processor, perform steps in a method on a Mobile Station (MS) comprising at least one camera and a depth sensor, the method comprising:

capturing a first image with the at least one camera, the first image comprising color information for at least a portion of an environment being modeled by the MS;

obtaining a camera pose for the first image;

determining, based, in part, on color information in the first image and the camera pose for the first image, whether to update a volumetric dataset associated with a first 3-Dimensional (3D) model of the environment; and

enabling the depth sensor to capture depth information when the first 3D model is updated, wherein the captured depth information is used, at least in part, to update the first 3D model using volumetric reconstruction.

25. The computer-readable medium of claim 24, wherein enabling the depth sensor comprises at least one of:

turning on power supplied to the depth sensor, or

enabling functionality related to computation of a depth image.

26. The computer-readable medium of claim 24, further comprising:

maintaining disablement of the depth sensor, when the first 3D model is not extended.

27. The computer-readable medium of claim 24, wherein a determination to update the volumetric dataset is made when:

color information in the first image includes data pertaining to one or more points in the environment, which are unrepresented in the volumetric dataset;

or

color information in the first image includes data pertaining to one or more points in the environment represented in the volumetric dataset and for which additional depth data is desired;

or

the first image is the last of a predetermined number of consecutive images with color information since the depth sensor was last enabled.

28. The computer-readable medium of claim 24, wherein upon enabling the depth sensor, the first 3D model is updated using volumetric reconstruction by:

capturing, with the at least one camera, a second image with color information for the environment;

augmenting the second image with the captured depth information provided by the depth sensor; and

fusing, based, in part, on a camera pose for the second image, the augmented second image with the volumetric dataset associated with the first 3D model.

29. The computer-readable medium of claim 28, wherein the volumetric dataset is represented using a Truncated Signed Distance Function (TDSF).

30. The computer-readable medium of claim 29, wherein fusing the augmented second image with the volumetric dataset comprises:

determining the camera pose for the second image;

projecting, based on the camera pose of the second image, one or more data samples in the TDSF into a depth map based on the augmented second image;

determining, for each of the one or more data samples projected into the depth map, a corresponding measured relative distance from a location of the projected data sample to a surface defined by the TDSF;

obtaining, for each of the one or more data samples projected into the depth map, a corresponding mapped TDSF value by mapping the corresponding measured relative distance to the TDSF; and

combining, for each of the one or more one or more data samples in the TDSF, the corresponding mapped TDSF value with a stored TDSF value for the data sample.