Using photometric stereo for 3D environment modeling

• US 9,857,470 B2
• Filed: 12/28/2012
• Issued: 01/02/2018
• Est. Priority Date: 12/28/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hilliges, Otmar, Weiss, Malte Hanno, Izadi, Shahram, Kim, David, Rother, Carsten Curt Eckard
• Primary Examiner(s): MCINNISH, KEVIN K
• Application Number:
  US13/729,324
• Publication Number:
  US 20140184749A1
• Time in Patent Office: 1,831 Days
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: H04N 13/20 Image signal generators

Abstract

Detecting material properties such reflectivity, true color and other properties of surfaces in a real world environment is described in various examples using a single hand-held device. For example, the detected material properties are calculated using a photometric stereo system which exploits known relationships between lighting conditions, surface normals, true color and image intensity. In examples, a user moves around in an environment capturing color images of surfaces in the scene from different orientations under known lighting conditions. In various examples, surfaces normals of patches of surfaces are calculated using the captured data to enable fine detail such as human hair, netting, textured surfaces to be modeled. In examples, the modeled data is used to render images depicting the scene with realism or to superimpose virtual graphics on the real world in a realistic manner.

Claims

1. A method at a 3D environment modeling system comprising:

at least one of accessing or generating a coarse 3D model of an environment, the coarse 3D model of the environment being generated in a 3D volume made up of a plurality of voxels and corresponding images stored on a memory device;

receiving a first image of the environment captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in the environment;

obtaining at a processor, camera pose of at least one camera at the mobile environment capture device used to capture the received first image;

determining whether a location in the memory device is available for the first image;

based on a determination that the location is not available, determining whether the first image is higher quality than a second image stored in the memory device for one of the plurality of voxels;

based on determining that the first image is higher quality than the second image stored in the memory device, replacing the second image stored in the memory device with the first image;

calculating a representation of fine detail of surfaces in the environment, the representation of fine detail comprising at least one true color of the environment, the calculation being based at least in part on at least one surface normal of at least one element in the environment, the first image, camera pose, and lighting conditions;

refining the coarse 3D model based at least in part on the representation of fine detail; and

causing a projection of at least a portion of the refined 3D model into real space.

2. A method as claimed in claim 1 the representation of fine detail comprising surface normal estimates at a resolution capable of depicting textured surfaces.

3. A method as claimed in claim 1 comprising using the representation of surfaces in the environment to either render an image depicting at least part of the surfaces or to superimpose virtual graphics over the environment.

4. A method as claimed in claim 1 further comprising receiving a plurality of images comprising depth images and color images of the environment captured by the mobile environment capture device moved in the unplanned manner in the environment.

5. A method as claimed in claim 1 comprising accessing a coarse 3D model of the environment and refining the coarse 3D model using the representation of fine detail of surfaces.

6. A method as claimed in claim 4 comprising using a data capture strategy to select first image for storage, the data capture strategy comprising using selection criteria related to one or more of the following:

distance of the mobile environment capture device to a surface in the environment, orientation of the mobile environment capture device with respect to a surface in the environment, relative position of a specified surface depicted in an image and the center of the image.

7. A method as claimed in claim 1 comprising:

storing a 3D model of the environment using the 3D volume, the plurality of voxels associated with the 3D volume being related to a position in the 3D environment being modeled; and

storing, at each voxel, a pointer to an array comprising pointers to memory locations storing camera pose and image data related to each voxel.

8. A method as claimed in claim 1 further comprising using the plurality of images to construct the coarse 3D environment model of the environment by aggregating information from the plurality of images.

9. A method as claimed in claim 8 further comprising refining the coarse 3D environment model by calculating the representation of fine detail for surface patches of surfaces depicted in the coarse 3D environment model.

10. A method as claimed in claim 1 further comprising obtaining the camera pose by one or more of the following:

sensed data from position and orientation sensors at the mobile environment capture device, and calculating the camera pose from images captured by the mobile environment capture device.

11. A 3D environment modeling system comprising:

a memory device;

an input interface arranged to receive a first of an environment captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in the environment;

the input interface further arranged to perform at least one of accessing or generating a coarse 3D model of the environment, the coarse 3D model of the environment being generated in a 3D volume made up of a plurality of voxels and corresponding images stored on the memory device;

a processor arranged to obtain camera pose of at least one camera at the mobile environment capture device used to capture the received first image;

the processor further arranged to;

determine whether a location in the memory device is available for the first image;

based on a determination that the location is not available, determine whether the first image is higher quality than a second image stored in the memory device for one of the plurality of voxels based at least in part on one or more of the following;

a distance from a voxel to the center of the first or the second image;

a distance between the camera and a surface depicted by the voxel; and

a relationship between the surface depicted by the voxel and a camera pose associated with the first or the second image; and

based on determining that the first image is higher quality than the second image, replacing the second image stored in the memory device with the first image; and

a photometric stereo system arranged to calculate a representation of fine detail of surfaces in the environment, the representation of fine detail comprising at least one true color of the environment, the calculation being based at least in part on at least one surface normal of at least one element in the environment, camera pose, the first image, and lighting conditions;

refining the coarse 3D model based at least in part on the representation of fine detail; and

a projection engine arranged to cause a projection of at least a portion of the refined 3D model into real space.

12. A system as claimed in claim 11 the photometric stereo system being at least partially implemented using hardware logic selected from one or more of the following:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.

13. An augmented reality system comprising a 3D environment modeling system as claimed in claim 11, the augmented reality system being arranged to superimpose virtual graphics over a real world environment, the virtual graphics taking into account the representation of fine detail of surfaces in the environment.

14. A system as claimed in claim 11, wherein the processor is further arranged to obtain the camera pose by one or more of the following:

sensed data from position and orientation sensors at the mobile environment capture device, and calculating the camera pose from images captured by the mobile environment capture device.

15. A computer-readable storage media storing device-executable instructions to:

perform at least one of accessing or generating a coarse 3D model of an environment, the coarse 3D model of the environment being generated in a 3D volume made up of a plurality of voxels and corresponding images stored on the memory device;

receive a first image of the environment captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in the environment;

obtain, at a processor, camera pose of at least one camera at the mobile environment capture device used to capture the first image;

determine whether a location in the memory device is available for the first image;

based on a determination that the location is not available, determine whether the first image is higher quality than a second image stored in the memory device for one of the plurality of voxels, based at least in part on comparing criteria of the first image to criteria of the second image, the criteria comprising one or more of the following;

a distance from a voxel to the center of the first image or the second image;

a distance between the camera and a surface depicted by the voxel; and

a relationship between the surface depicted by the voxel and a camera pose associated with the first image or the second image;

based on determining that the first image is higher quality than the second image, replacing the second image stored in the memory device with the first image;

calculate a representation of fine detail of surfaces in the environment, the representation of fine detail comprising at least one true color of the environment, the calculation being based at least in part on at least one surface normal of at least one element in the environment, camera pose, the first image, and lighting conditions;

refine the coarse 3D model based at least in part on the representation of fine detail; and

cause a projection of at least a portion of the refined 3D model into real space.

16. A computer-readable storage media as claimed in claim 15 further comprising device-executable instructions to use the representation of surfaces in the environment to either render an image depicting at least part of the surfaces or to superimpose virtual graphics over the environment.

17. A computer-readable storage media as claimed in claim 15 further comprising device-executable instructions to receive a plurality of images comprising depth images and color images.

18. A computer-readable storage media as claimed in claim 15 further comprising device-executable instructions to:

store a 3D model of the environment using the 3D volume represented in memory, the plurality of voxels associated with the 3D volume being related to a position in the 3D environment being modeled; and

store, at each voxel, a pointer to an array comprising pointers to memory locations storing camera pose and image data related to each voxel.

19. A computer-readable storage media as claimed in claim 17 further comprising device-executable instructions to construct the coarse 3D environment model of the environment by aggregating information from the plurality of images.

20. A computer-readable storage media as claimed in claim 15 further comprising device-executable instructions to obtain the camera pose by one or more of the following:

sensed data from position and orientation sensors at the mobile environment capture device, and calculating the camera pose from images captured by the mobile environment capture device.