Using photometric stereo for 3D environment modeling
First Claim
1. A method at a 3D environment modeling system comprising:
- at least one of accessing or generating a coarse 3D model of an environment, the coarse 3D model of the environment being generated in a 3D volume made up of a plurality of voxels and corresponding images stored on a memory device;
receiving a first image of the environment captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in the environment;
obtaining at a processor, camera pose of at least one camera at the mobile environment capture device used to capture the received first image;
determining whether a location in the memory device is available for the first image;
based on a determination that the location is not available, determining whether the first image is higher quality than a second image stored in the memory device for one of the plurality of voxels;
based on determining that the first image is higher quality than the second image stored in the memory device, replacing the second image stored in the memory device with the first image;
calculating a representation of fine detail of surfaces in the environment, the representation of fine detail comprising at least one true color of the environment, the calculation being based at least in part on at least one surface normal of at least one element in the environment, the first image, camera pose, and lighting conditions;
refining the coarse 3D model based at least in part on the representation of fine detail; and
causing a projection of at least a portion of the refined 3D model into real space.
2 Assignments
0 Petitions
Accused Products
Abstract
Detecting material properties such reflectivity, true color and other properties of surfaces in a real world environment is described in various examples using a single hand-held device. For example, the detected material properties are calculated using a photometric stereo system which exploits known relationships between lighting conditions, surface normals, true color and image intensity. In examples, a user moves around in an environment capturing color images of surfaces in the scene from different orientations under known lighting conditions. In various examples, surfaces normals of patches of surfaces are calculated using the captured data to enable fine detail such as human hair, netting, textured surfaces to be modeled. In examples, the modeled data is used to render images depicting the scene with realism or to superimpose virtual graphics on the real world in a realistic manner.
-
Citations
20 Claims
-
1. A method at a 3D environment modeling system comprising:
-
at least one of accessing or generating a coarse 3D model of an environment, the coarse 3D model of the environment being generated in a 3D volume made up of a plurality of voxels and corresponding images stored on a memory device; receiving a first image of the environment captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in the environment; obtaining at a processor, camera pose of at least one camera at the mobile environment capture device used to capture the received first image; determining whether a location in the memory device is available for the first image; based on a determination that the location is not available, determining whether the first image is higher quality than a second image stored in the memory device for one of the plurality of voxels; based on determining that the first image is higher quality than the second image stored in the memory device, replacing the second image stored in the memory device with the first image; calculating a representation of fine detail of surfaces in the environment, the representation of fine detail comprising at least one true color of the environment, the calculation being based at least in part on at least one surface normal of at least one element in the environment, the first image, camera pose, and lighting conditions; refining the coarse 3D model based at least in part on the representation of fine detail; and causing a projection of at least a portion of the refined 3D model into real space. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A 3D environment modeling system comprising:
-
a memory device; an input interface arranged to receive a first of an environment captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in the environment; the input interface further arranged to perform at least one of accessing or generating a coarse 3D model of the environment, the coarse 3D model of the environment being generated in a 3D volume made up of a plurality of voxels and corresponding images stored on the memory device; a processor arranged to obtain camera pose of at least one camera at the mobile environment capture device used to capture the received first image; the processor further arranged to; determine whether a location in the memory device is available for the first image; based on a determination that the location is not available, determine whether the first image is higher quality than a second image stored in the memory device for one of the plurality of voxels based at least in part on one or more of the following; a distance from a voxel to the center of the first or the second image; a distance between the camera and a surface depicted by the voxel; and a relationship between the surface depicted by the voxel and a camera pose associated with the first or the second image; and based on determining that the first image is higher quality than the second image, replacing the second image stored in the memory device with the first image; and a photometric stereo system arranged to calculate a representation of fine detail of surfaces in the environment, the representation of fine detail comprising at least one true color of the environment, the calculation being based at least in part on at least one surface normal of at least one element in the environment, camera pose, the first image, and lighting conditions; refining the coarse 3D model based at least in part on the representation of fine detail; and a projection engine arranged to cause a projection of at least a portion of the refined 3D model into real space. - View Dependent Claims (12, 13, 14)
-
-
15. A computer-readable storage media storing device-executable instructions to:
-
perform at least one of accessing or generating a coarse 3D model of an environment, the coarse 3D model of the environment being generated in a 3D volume made up of a plurality of voxels and corresponding images stored on the memory device; receive a first image of the environment captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in the environment; obtain, at a processor, camera pose of at least one camera at the mobile environment capture device used to capture the first image; determine whether a location in the memory device is available for the first image; based on a determination that the location is not available, determine whether the first image is higher quality than a second image stored in the memory device for one of the plurality of voxels, based at least in part on comparing criteria of the first image to criteria of the second image, the criteria comprising one or more of the following; a distance from a voxel to the center of the first image or the second image; a distance between the camera and a surface depicted by the voxel; and a relationship between the surface depicted by the voxel and a camera pose associated with the first image or the second image; based on determining that the first image is higher quality than the second image, replacing the second image stored in the memory device with the first image; calculate a representation of fine detail of surfaces in the environment, the representation of fine detail comprising at least one true color of the environment, the calculation being based at least in part on at least one surface normal of at least one element in the environment, camera pose, the first image, and lighting conditions; refine the coarse 3D model based at least in part on the representation of fine detail; and cause a projection of at least a portion of the refined 3D model into real space. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification