USING PHOTOMETRIC STEREO FOR 3D ENVIRONMENT MODELING

• US 20140184749A1
• Filed: 12/28/2012
• Published: 07/03/2014
• Est. Priority Date: 12/28/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hilliges, Otmar, Weiss, Malte Hanno, Izadi, Shahram, Kim, David, Rother, Carsten Curt Eckard
• Granted Patent: US 9,857,470 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: H04N 13/20 Image signal generators

Abstract

Detecting material properties such reflectivity, true color and other properties of surfaces in a real world environment is described in various examples using a single hand-held device. For example, the detected material properties are calculated using a photometric stereo system which exploits known relationships between lighting conditions, surface normals, true color and image intensity. In examples, a user moves around in an environment capturing color images of surfaces in the scene from different orientations under known lighting conditions. In various examples, surfaces normals of patches of surfaces are calculated using the captured data to enable fine detail such as human hair, netting, textured surfaces to be modeled. In examples, the modeled data is used to render images depicting the scene with realism or to superimpose virtual graphics on the real world in a realistic manner.

Claims

1. A mobile capture device, comprising :

a housing;

a high brightness point light source fixed at the housing at a known position;

at least one image capture device supported by the housing and arranged to capture a plurality of color images as the device moves in an environment, the images depicting surfaces in the environment from different orientations and illuminated by the light source;

at least one sensor arranged to capture data to track location and orientation of the device associated with each color image;

a communications interface arranged to transfer images and data captured by the device to a photometric stereo system.

2. A device as claimed in claim 1 which is hand-held.

3. A device as claimed in claim 1 the at least one image capture device arranged to capture depth images using an optical axis which is aligned with an optical axis of the image capture device used to capture the plurality of color images.

4. A device as claimed in claim 1 comprising two image capture devices, one being a color camera and one being a depth camera.

5. A device as claimed in claim 4 comprising optical axis alignment equipment arranged to align the optical axes of the two image capture devices.

6. A device as claimed in claim 1 comprising an infra-red illumination source.

7. A device as claimed in claim 1 the high brightness point light source comprising a light emitting diode and a heat sink.

8. A method at a 3D environment modeling system comprising:

receiving images captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in an environment;

obtaining at a processor, camera pose of at least one camera at the mobile environment capture device used to capture the received images; and

calculating, using a relationship between the images, camera pose and lighting conditions, a representation of fine detail of surfaces in the environment.

9. A method as claimed in claim 8 the representation of fine detail comprising surface normal estimates at a resolution capable of depicting textured surfaces.

10. A method as claimed in claim 8 comprising using the representation of surfaces in the environment to either render an image depicting at least part of the surfaces or to superimpose virtual graphics over the environment.

11. A method as claimed in claim 8 wherein the received images comprise depth images and color images.

12. A method as claimed in claim 8 comprising accessing a coarse 3D model of the environment and refining the coarse 3D model using the representation of fine detail of surfaces.

13. A method as claimed in claim 8 comprising using a data capture strategy to select a plurality of the received images for storage, the data capture strategy comprising using selection criteria related to any one or more of:

distance of the mobile environment capture device to a surface in the environment, orientation of the mobile environment capture device with respect to a surface in the environment, relative position of a specified surface depicted in an image and the center of the image.

14. A method as claimed in claim 8 comprising storing a 3D model of the environment using a volume represented in memory, the volume comprising a plurality of voxels each related to a position in the 3D environment being modeled, and storing at each voxel a pointer to an array comprising pointers to memory locations storing camera pose and image data related to the voxel.

15. A method as claimed in claim 8 comprising using a plurality of the received images to construct a coarse 3D environment model of the environment by aggregating information from the plurality of images.

16. A method as claimed in claim 15 comprising refining the coarse 3D environment model by calculating the representation of fine detail for surface patches of surfaces depicted in the coarse 3D environment model.

17. A method as claimed in claim 8 comprising obtaining the camera pose by any one or more of:

sensed data from position and orientation sensors at the mobile environment capture device, and calculating the camera pose from images captured by the mobile environment capture device.

18. A 3D environment modeling system comprising:

an input interface arranged to receive images captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in an environment;

a processor arranged to obtain camera pose of at least one camera at the mobile environment capture device used to capture the received images; and

a photometric stereo system arranged to calculate, using a relationship between the images, camera pose and lighting conditions, a representation of fine detail of surfaces in the environment.

19. A system as claimed in claim 18 the photometric stereo system being at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.

20. An augmented reality system comprising a 3D environment modeling system as claimed in claim 18, the augmented reality system being arranged to superimpose virtual graphics over the real world, the virtual graphics taking into account the representation of fine detail of surfaces in the environment.