Method and system for 3D capture based on structure from motion with simplified pose detection

• US 10,068,344 B2
• Filed: 03/05/2015
• Issued: 09/04/2018
• Est. Priority Date: 03/05/2014
• Status: Active Grant

• Current Assignee: Smart Picture Technologies Inc.
• Original Assignee: Smart Picture Technologies Inc.
• Inventors: Jovanovich, Dejan, Beardmore, Keith, Myllykoski, Kari, Grotelueschen, James H.
• Primary Examiner(s): Liew, Alex
• Application Number:
  US15/123,662
• Publication Number:
  US 20170337701A1
• Time in Patent Office: 1,279 Days
• Field of Search: 382106, 382154
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

Aspects disclosed herein provide methods, systems, and devices for 3D image capture based on a Structure from Motion (SFM) imaging technique with simplified pose detection. The present disclosure provides a straightforward method to directly track a camera'"'"'s motion (pose detection) thereby removing a substantial portion of the computing load needed to build a dimensionally accurate 3D model from a sequence of images taken by the camera.

Claims

1. An integrated 3D modeling device comprising:

a camera configured to capture a sequence of images from a sequence of poses relative to a known reference template pattern placed into a scene, or relative to known reference object(s) in the scene to be modeled;

a template pattern recognition engine configured to recognize the known reference template pattern placed in the scene to be modeled and determine a position of known points of the known reference template pattern or known reference object in an image from the sequences of images captured by the camera;

a pose engine configured to determine an orientation and a position of the camera (camera pose) relative to the known reference template pattern or known reference object recognized by the template pattern recognition engine;

a SFM (Structure from Motion) engine configured to determine a relative displacement of each pixel (pixel displacement) from one image to another image from the sequence of images using optical flow, and determine a range to points representing features in the scene from a viewpoint of the camera that captured the images using the pixel displacement produced by the SFM engine and camera pose calculated by the pose engine; and

a 3D modeling engine configured to combine product from the template pattern recognition and pose engines with product from the SFM engine to provide a dimensionally correct 3D map output of the scene used to generate a 3D model.

2. The 3D modeling device of claim 1, wherein the known reference template pattern has at least four co-planar reference fiducials.

3. The 3D modeling device of claim 1, wherein the known reference template pattern comprises a printed pattern.

4. The 3D modeling device of claim 3, wherein the known reference template pattern comprises a sticker which can be affixed to objects in the scene.

5. The 3D modeling device of claim 3, wherein the known reference template pattern includes color(s) which can more easily be extracted via a digital filter.

6. The 3D modeling device of claim 1, wherein the known reference template pattern includes a pattern of electromagnetic emitting elements.

7. The 3D modeling device of claim 1, wherein the known template pattern recognition engine is further configured to recognize a series of reference template patterns that are placed into the scene or near the object to be modeled, and determine a position of known points of each known reference template pattern in the series.

8. The 3D modeling device of claim 3, wherein the printed pattern comprises at least four co-planar reference fiducials.

9. The 3D modeling device of claim 1, wherein the 3D modeling engine comprises a data processing system configured to create a Quantified Image or Data File which includes the 3D model of the scene and automatically generated measurement data.

10. The 3D modeling device of claim 1, wherein the sequence of poses is fully or partially achieved by moving the object(s) to be modeled.

11. The 3D modeling device of claim 1, wherein the 3D modeling engine is configured to detect and reject outlier data to find a model that best fits a collection of data.

12. The 3D modeling device of claim 11, wherein the 3D modeling engine is further configured to use data filtering to reject the outlier data to find the model that best fits the collection of data.

13. The 3D modeling device of claim 1, wherein the user can query the 3D modeling device to receive specific dimensional measurements from the 3D model.

14. The 3D modeling device of claim 13, wherein the 3D model is stored together with specific measurement data automatically extracted and measurement data of parameters which were manually identified from the 3D model in a Quantified Image or Data File.

15. The 3D modeling device of claim 1, wherein the dimensionally correct 3D model of the scene are in a world coordinate system with an origin that is a specific point relative to the known reference template pattern.

16. The 3D modeling device of claim 15, wherein when more than one known reference template patterns or reference objects is used, the origin of the world coordinate system is a specific point relative to a first known template pattern or reference object detected.

17. The 3D modeling device of claim 1, wherein the 3D modeling engine is further configured to monitor the dimensionally correct 3D model of the scene to estimate an accuracy of the 3D model as more data is captured.

18. The 3D modeling device of claim 17, wherein a user is notified that an estimate accuracy threshold of the 3D model has been reached.

19. The 3D modeling device of claim 1, further comprising a Data Storage Unit configured to store data comprising application specific data, a Quantified Image or Data File, and/or the 3D model.