Optimized object scanning using sensor fusion

• US 10,257,505 B2
• Filed: 02/08/2016
• Issued: 04/09/2019
• Est. Priority Date: 02/08/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Scavezze, Michael, Tomlin, Arthur, Cai, Rui, Li, Zhiwei
• Primary Examiner(s): Perungavoor, Sathyanaraya V
• Assistant Examiner(s): Brown, Jr., Howard D
• Application Number:
  US15/017,799
• Publication Number:
  US 20170230641A1
• Time in Patent Office: 1,156 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Sensor fusion is utilized in an electronic device such as a head mounted display (HMD) device that has a sensor package equipped with different sensors so that information that is supplemental to captured 2D images of objects or scenes in a real world environment may be utilized to determine an optimized transform of image stereo-pairs and to discard erroneous data that would otherwise prevent successful scans used for construction of a 3D model in, for example, virtual world applications. Such supplemental information can include one or more of world location, world rotation, image data from an extended field of view (FOV), or depth map data.

Claims

1. A method of capturing images of an object from which to construct a three-dimensional model of the object, the method comprising:

using an image capture device, capturing a plurality of images of the object in a physical environment from a plurality of vantage points;

identifying points in the plurality of images that are consistently located across all images as background points;

filtering points in the plurality of images to remove background points from use in construction of the three-dimensional model of the object;

using at least one sensor associated with the image capture device, detecting information about one or more of;

motion or pose of the image capture device;

selecting individual ones of the captured plurality of images on the basis of the detected information; and

sending the selected individual ones of the captured plurality of images to a process for constructing the three-dimensional model of the object.

2. The method of claim 1 further comprising selecting the individual ones of the plurality of images based on a detected position or orientation or pose of the image capture device being nearest to one or more predetermined poses of the image capture device.

3. The method of claim 1 further comprising selecting the individual ones of the plurality of images based on a detected motion of the image capture device, associated with capture of the individual ones of the plurality of images, being at or below a predetermined threshold.

4. The method of claim 1 further comprising selecting a subset of the plurality of images based on a detected motion of the image capture device being at or below a predetermined threshold.

5. The method of claim 1 further comprising:

exposing a user interface in communication with the image capture device that is configured to guide a user of the image capture device to control one or more of the position, motion, or orientation of the image capture device to facilitate the capture of images from which to construct the three-dimensional model of the object.

6. The method of claim 5 further comprising

using data from the at least one sensor associated with the image capture device to guide the user.

7. The method of claim 1 further comprising:

examining the filtered points to determine positional discrepancies in features of the object that are inconsistent with the detected position, motion, or orientation of the image capture device within the physical environment contemporaneously with the capture of each of the plurality of images;

segmenting the plurality of images into groups divided by the appearance of the positional discrepancies; and

resolving the positional discrepancies to form a coherent computational model of the object based on the filtered points having background points removed.

8. A device operative to perform object scanning using sensor fusion, comprising:

an outward-facing image sensor operative to capture images

at least one sensor operative to detect one or more of a position, motion, or orientation of the device within the space, wherein the at least one sensor comprises one of tracking camera, inertia sensor, magnetic 6-degrees-of-freedom position sensor, a lighthouse-based laser-scanning system, and synchronized photodiodes on the object being tracked;

one or more processors;

a data storage system, operative to store images from the outward-facing image sensor, and to store position, motion, or orientation data from the at least one sensor; and

a non-transitory machine-readable memory device operative to store instructions, which when executed cause the one or more processors to

capture a plurality of images of the scene from respective positions within the space,

detect a position, motion, or orientation of the device within the space associated with the capture of each of the plurality of images of the scene, and

discard one or more of the plurality of captured images based on the detected position, motion, or orientation of the device at a respective capture location.

9. The device of claim 8 wherein the outward facing image sensor comprises at least one of a two-dimensional image sensor, a stereoscopic image sensor, and a depth sensor.

10. The device of claim 8 further comprising a user interface providing at least one of an auditory, visual, or haptic feedback to a user and being responsive to verbal, tactile, or gestural input by the user.

11. The device of claim 8 as incorporated in a head mounted display device.

12. The device of claim 8 wherein the instructions cause the one or more processors to construct a three-dimensional model using captured images other than the discarded images.

13. The device of claim 8 wherein the instructions cause the one or more processors to utilize one or more of the detected position, motion, or orientation of the device as an initial condition for determining a transform of the captured images.

14. The device of claim 8 further comprising an extended field of view (FOV) image sensor having an FOV that exceeds the image capture sensor in which the extended FOV image sensor is configured to determine poses of the image capture sensor in which to capture images suitable for constructing a three-dimensional model of the object.

15. A non-transitory machine-readable memory device operative to store instructions which, when executed by one or more processors disposed in an electronic device, cause the electronic device to:

perform object scanning by capturing a plurality of images of an object from a respective plurality of vantage points using a first camera disposed in the electronic device;

determine object poses for the scanning using a second camera disposed in the electronic device that has an extended field of view relative to the first camera;

generate world tracking metadata for the electronic device at each vantage point; and

utilize the world tracking metadata to combine a subset of the plurality of captured images into a three-dimensional model of the object,

in which the first camera has higher angular resolution or is configured to capture an increased level of detail relative to the second camera and the tracking metadata is generated using one or more of tracking camera or inertia sensor incorporated in the electronic device.

16. The non-transitory machine-readable memory device of claim 15 in which the instructions cause the electronic device to generate depth maps from captured images for each vantage point.

17. The non-transitory machine-readable memory device of claim 16 in which the instructions cause the electronic device to operate the first camera to capture images at evenly-spaced vantage points so as to minimize differences in error characteristics in the generated depth maps.