Capturing and aligning three-dimensional scenes

• US 9,324,190 B2
• Filed: 02/25/2013
• Issued: 04/26/2016
• Est. Priority Date: 02/24/2012
• Status: Active Grant

• Current Assignee: Matterport, Inc
• Original Assignee: Matterport, Inc
• Inventors: Gausebeck, David, Beebe, Michael, Bell, Matthew
• Primary Examiner(s): Elahi, Shan
• Application Number:
  US13/776,688
• Publication Number:
  US 20140043436A1
• Time in Patent Office: 1,156 Days
• Field of Search: 348/46
• US Class Current: 1/1
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

Systems and methods for building a three-dimensional composite scene are disclosed. Certain embodiments of the systems and methods may include the use of a three-dimensional capture device that captures a plurality of three-dimensional images of an environment. Some embodiments may further include elements concerning aligning and/or mapping the captured images. Various embodiments may further include elements concerning reconstructing the environment from which the images were captured. The methods disclosed herein may be performed by a program embodied on a non-transitory computer-readable storage medium when executed the program is executed a processor.

Claims

1. A three-dimensional image reconstruction system, comprising:

a three-dimensional capture device comprising;

a housing;

a sensor module comprising one or more sensors respectively configured to capture three-dimensional image data captures of an object or environment at different capture positions relative to a three-dimensional coordinate space during a capture process, wherein the three-dimensional image data captures are respectively associated with distance information for features included therein; and

a communication module configured to communicatively couple the sensor module to a mobile processing device and send the three-dimensional image data captures, or sets of three-dimensional data respectively derived from the three-dimensional image data captures, to a mobile processing device as they are respectively captured or derived in association with the capture process, wherein the mobile processing device is configured to iteratively align the three-dimensional image data captures or the sets, based on the distance information, as the three-dimensional image data captures or the sets are respectively received to generate three-dimensional representations of the object or environment, and wherein the mobile processing device is configured to render the three-dimensional representations of the object or environment via a display screen of the mobile processing device.

2. The system of claim 1, wherein in response to reception of the three-dimensional image data captures, the mobile processing device is configured to derive the distance information for the features respectively included therein, and align the features to one another in the three-dimensional coordinate space based on the distance information to generate the three-dimensional representations of the object or environment concurrently with the capture process.

3. The system of claim 1, wherein the sets of the three-dimensional data respectively comprise the distance information derived for respective features included in the three-dimensional image data captures, and wherein in response to reception of the sets, the mobile processing device is configured to align the respective features included in the sets to one another relative to the three-dimensional coordinate space based on the distance information for the respective features to generate the three-dimensional representations of the object or environment and render three-dimensional representations of the object or environment concurrently with the capture process.

4. The system of claim 1, wherein the communication module comprises a transmitter configured to wirelessly transmit the image data captures or the sets of the three-dimensional data to the mobile processing device.

5. The system of claim 1, wherein the three-dimensional capture device further comprises an attachment component configured to physically couple the three-dimensional capture device to the mobile processing device.

6. The system of claim 5, wherein the attachment component comprises one or more clips configured to attach or detach the three-dimensional capture device to or from the mobile processing device.

7. The system of claim 5, wherein the attachment component comprises a molded docking station having a section that is an inverse shape of a section of the mobile processing device, wherein the respective inverse sections mate to securely couple the devices.

8. The system of claim 5, wherein the attachment component comprises hardware configured to couple a data port of the mobile processing device to the three-dimensional capture device, wherein the hardware provides physical support to stabilize and fix the mobile processing device in an attached position relative to the three-dimensional capture device.

9. The system of claim 1, further comprising a battery configured to power operation of the three-dimensional capture device.

10. The system of claim 1, wherein the three-dimensional capture device, further comprises:

a memory to store computer executable instructions, and

a processor, coupled to the memory, that facilitates execution of the computer executable instructions to perform operations, comprising;

determining the distance information for the features respectively included in the three-dimensional image data captures; and

deriving the three-dimensional data sets from the three-dimensional image data captures.

11. The system of claim 1, wherein the sensor module comprises two or more sensors respectively configured to capture different ones of the three-dimensional image data captures, and wherein the two or more sensors are respectively positioned at different angles.

12. The system of claim 1, wherein the mobile processing device is at least one of:

a mobile phone, a tablet computer, or a wearable computer.

13. The system of claim 1, wherein the three-dimensional capture device and the mobile processing device are configured to be communicatively connected via a cable.

14. The three-dimensional image reconstruction system of claim 1, wherein the one or more sensors are respectively configured to capture the three-dimensional image data captures of the object or environment at different orientations relative to the three-dimensional coordinate space during the capture process.

15. The three-dimensional image reconstruction system of claim 1, wherein at least two of the three-dimensional image data captures or sets of three-dimensional data comprise no overlapping features.

16. The three-dimensional image reconstruction system of claim 1, wherein the three-dimensional representations of the object or environment comprise a point cloud, a three-dimensional mesh, a volumetric rendering, a surfel cloud, or a cartoon rendering.

17. A three-dimensional capture device, comprising:

a housing comprising a handle;

one or more sensors respectively configured to capture three-dimensional image data captures of an object or environment at different capture positions relative to a three-dimensional coordinate space over a capture process, wherein the three-dimensional image data captures are respectively associated with distance information for features included therein; and

a communication module configured to send the three-dimensional image data captures, or sets of three-dimensional information respectively derived from the three-dimensional image data captures, to a mobile device as they are respectively captured or derived, wherein the mobile device is configured to iteratively align the three-dimensional image data captures or the sets, as the three-dimensional image data captures or the sets are respectively received, based on the distance information, to generate three-dimensional representations of the object or environment, and wherein the mobile device is configured to render the three-dimensional representations of the object or environment via a display screen of the mobile device.

18. The three-dimensional capture device of claim 17, wherein in response to reception of the three-dimensional image data captures, the mobile device is configured to derive the distance information for the features respectively included therein, and align the features to one another in the three-dimensional coordinate space based on the distance information to generate the three-dimensional representations of the object or environment concurrently with the capture process.

19. The three-dimensional capture device of claim 17, wherein the sets of the three-dimensional information respectively comprise the distance information derived for respective features included in the three-dimensional image data captures, and wherein in response to reception of the sets, the mobile device is configured to align the respective features included in the sets to one another relative to the three-dimensional coordinate space based on the distance information for the respective features to generate the three-dimensional representations of the object or environment and render three-dimensional representations of the object or environment concurrently with the capture process.

20. The three-dimensional capture device of claim 17, wherein the housing comprises a pole shape with a first end and a second, wherein the handle is positioned at the first end of the pole shape, and wherein the one or more sensors are positioned at the second end of the pole shape.

21. The three-dimensional capture device of claim 17, wherein the housing comprises a curved body formed around at least a portion of the handle and wherein the one or more sensors include a plurality of sensors located at different positions on a surface of the curved body at various angles relative to the handle.

22. The three-dimensional capture device of claim 17, wherein the one or more sensors include a plurality of sensors located on the housing and positioned at different angles relative to the handle such that respective fields of view of adjacent sensors of the plurality of sensors overlap.

23. The three-dimensional capture device of claim 17, further comprising:

a memory to store computer executable instructions, and

a processor, coupled to the memory, that facilitates execution of the computer executable instructions to perform operations, comprising;

determining the distance information for the features respectively included in the three-dimensional image data captures; and

deriving the three-dimensional data sets from the three-dimensional image data captures.

24. The three-dimensional capture device of claim 17, wherein the mobile device is at least one of:

a mobile phone, a tablet computer, a wearable device, or a handheld device.

25. The three-dimensional capture device of claim 17, further comprising a port to receive a cable for communicative coupling to the mobile device.

26. The three-dimensional capture device of claim 17, wherein the communication module comprises a transmitter configured to wirelessly transmit the three-dimensional image data captures or the sets of three-dimensional information respectively derived from the three-dimensional image data to the mobile device.

27. The three-dimensional capture device of claim 17, wherein the one or more sensors are respectively configured to capture three-dimensional image data captures of the object or environment at the different capture positions relative to the three-dimensional coordinate space based on movement of the handle over a capture process.

28. A non-transitory computer-readable storage medium comprising computer-readable instructions that, in response to execution, cause a computing system to perform operations, comprising:

receiving, from a three-dimensional capture device, three-dimensional image captures of an object or environment captured by the three-dimensional capture device at different positions relative to a three-dimensional coordinate space during a capture process, wherein the three-dimensional image captures are respectively associated with for features included therein;

aligning the three-dimensional image captures as they are received based in part on the distance information to generate three-dimensional representations of the object or environment; and

rendering the three-dimensional representations of the object or environment on a display.

29. The non-transitory computer-readable storage medium of claim 28, the operations further comprising:

deriving the distance information for the features respectively included in the three-dimensional image captures, wherein the aligning further comprises aligning the features to one another in the three-dimensional coordinate space based on the distance information concurrently with the capture process.

30. The non-transitory computer-readable storage medium of claim 28, wherein the rendering comprises rendering the three-dimensional representations of the object or environment as at least one of:

a point cloud, a three-dimensional mesh, a volumetric rendering, or a surfel cloud.

31. The non-transitory computer-readable storage medium of claim 28, the operations further comprising:

generating visual or audible feedback regarding at least one of;

aspects of the object or environment that have not been captured, or alignment errors associated with the three-dimensional representations of the object or environment.

32. The non-transitory computer-readable storage medium of claim 28, the operations further comprising:

receiving input identifying a perspective for viewing a three-dimensional representation of the object or environment as rendered via the display; and

rendering the three-dimensional representation of the object or environment in accordance with the perspective.

33. A mobile device, comprising:

a non-transitory computer readable medium that stores computer executable components;

a processor that executes at least the following computer executable components stored in the non-transitory computer readable medium;

a communication component configured to receive three-dimensional image captures of an object or environment from a three-dimensional capture device captured by the three-dimensional capture device at different positions relative to a three-dimensional coordinate space during a capture process, wherein the image captures are respectively associated with distance data for features included therein, and send the three-dimensional image captures to a remote processing system as they are respectively received, and receive three-dimensional representations of the object or environment from the remote processing system, wherein the remote processing system is configured to align the three-dimensional image captures as they are received based on the distance data for the features included therein to generate the three-dimensional representations of the object or environment; and

a presentation component configured to display the three-dimensional representations of the object or environment via a display of the mobile device in response reception of the three-dimensional representations.

34. The mobile device of claim 33, wherein the presentation component is configured to render the three-dimensional representations of the object or environment concurrently with the capture process.

35. The mobile device of claim 33, wherein the three-dimensional capture device is physically attached to the mobile device via an attachment mechanism that enables the three-dimensional capture device to be detached from the mobile device.