REAL-TIME THREE-DIMENSIONAL RECONSTRUCTION OF A SCENE FROM A SINGLE CAMERA

• US 20150279083A1
• Filed: 03/26/2014
• Published: 10/01/2015
• Est. Priority Date: 03/26/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Pradeep, Vivek, Rhemann, Christoph, Izadi, Shahram, Zach, Christopher, Bleyer, Michael, Bathiche, Steven
• Granted Patent: US 9,779,508 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

A combination of three computational components may provide memory and computational efficiency while producing results with little latency, e.g., output can begin with the second frame of video being processed. Memory usage may be reduced by maintaining key frames of video and pose information for each frame of video. Additionally, only one global volumetric structure may be maintained for the frames of video being processed. To be computationally efficient, only depth information may be computed from each frame. Through fusion of multiple depth maps from different frames into a single volumetric structure, errors may average out over several frames, leading to a final output with high quality.

Claims

1. A device comprising:

a camera tracking circuit having an input that receives an image from a sequence of images and an output providing a pose for the image, and an output storing the image as a key frame based on the pose of the image and a pose of other stored key frames;

a depth map estimation circuit having an input the receives the image from the sequence of images and one or more key frames from among the stored key frames selected for the image and having an output providing a depth map for the image; and

a volumetric fusion circuit having an input receiving the depth map for the image and an output providing a three-dimensional model as a fusion of depth maps received for the sequence of images.

2. The device of claim 1, wherein the camera tracking circuit, the depth map estimation circuit, and the volumetric fusion circuit are implemented within a single general purpose integrated circuit.

3. The device of claim 1, wherein the depth map for an image comprises a measure of depth for each pixel in the image.

4. The device of claim 1, wherein the pose for an image comprises rotation and translation of a camera with respect to a fixed coordinate system.

5. The device of claim 1, wherein the three-dimensional model is defined in a virtual volume, wherein an initial pose of a camera is defined at a point in the virtual volume.

6. The device of claim 1, wherein the three-dimensional model is defined by representing a surface using a signed distance field in the virtual volume.

7. The device of claim 1, wherein the three-dimensional model is output after processing a first and second image from the image sequence.

8. A process for generating a three-dimensional model from a sequence of images from a camera, the process comprising:

receiving an image from the sequence of images;

selecting one or multiple key frames from the sequence of images corresponding to the received image;

determining a depth map for the received image using the received image and the selected key frame(s);

merging the depth map into a volumetric representation of a scene represented in the image; and

repeating the receiving, selecting, determining and merging for successive images in the sequence of images.

9. The process of claim 8, further comprising determining a scale of depth by a sensor.

10. The process of claim 8, wherein the depth map for an image comprises a measure of depth for each pixel in the image.

11. The process of claim 8, wherein the pose for an image comprises rotation and translation of the camera with respect to a fixed coordinate system.

12. The process of claim 8, wherein the three-dimensional model is defined in a virtual volume, wherein an initial pose of the camera is defined at a point in the virtual volume.

13. The process of claim 8, wherein the three-dimensional model is defined by representing a surface using a signed distance field in the virtual volume.

14. The process of claim 8, wherein the three-dimensional model is output after processing a first and second image from the image sequence.

15. A computer program product, comprising:

a computer storage device;

computer program instructions stored in the computer storage device that when read from the storage device and processed by a processor of a computer instruct the computer to perform a process for generating a three-dimensional model from a sequence of images from a camera, the process comprising;

receiving an image from the sequence of images;

selecting a key frame from the sequence of images corresponding to the received image;

determining a depth map for the received image using the received image and the selected key frame;

merging the depth map into a volumetric representation of a scene represented in the image; and

repeating the receiving, selecting, determining and merging for successive images in the sequence of images.

16. The computer program product of claim 15, wherein the one or more processing units are one or more processing units housed in a mobile device that incorporates the camera.

17. The computer program product of claim 15, wherein the depth map for an image comprises a measure of depth for each pixel in the image.

18. The computer program product of claim 15, wherein the pose for an image comprises rotation and translation of the camera with respect to a fixed coordinate system.

19. The computer program product of claim 15, wherein the three-dimensional model is defined in a virtual volume, wherein an initial pose of the camera is defined at a point in the virtual volume.

20. The computer program product of claim 15, wherein the three-dimensional model is defined by representing a surface using a signed distance field in the virtual volume.