SYSTEMS AND METHODS FOR SCANNING THREE-DIMENSIONAL OBJECTS

• US 20170372527A1
• Filed: 06/22/2017
• Published: 12/28/2017
• Est. Priority Date: 06/22/2016
• Status: Active Grant

• Current Assignee: Packsize LLC (Packsize International, LLC)
• Original Assignee: Aquifi, Inc.
• Inventors: Murali, Giridhar, Blackstone, Keith, Peruch, Francesco, Dal Mutto, Carlo
• Granted Patent: US 10,311,648 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 17/00 Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
• CPC Class Codes: G01S 17/48 Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/337 : involving reference images or patches
• CPC Class Codes: G06T 7/38 : Registration of image sequences
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/254 : in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

A method for computing a three-dimensional (3D) model of an object includes: receiving, by a processor, a first chunk including a 3D model of a first portion of the object, the first chunk being generated from a plurality of depth images of the first portion of the object; receiving, by the processor, a second chunk including a 3D model of a second portion the object, the second chunk being generated from a plurality of depth images of the second portion of the object; computing, by the processor, a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks; aligning, by the processor, the first chunk with the second chunk in accordance with the registration; and outputting, by the processor, a 3D model corresponding to the first chunk merged with the second chunk.

Claims

1. A method for computing a three-dimensional (3D) model of an object, the method comprising:

receiving, by a processor, a first chunk comprising a 3D model of a first portion of the object, the first chunk being generated from a plurality of depth images of the first portion of the object;

receiving, by the processor, a second chunk comprising a 3D model of a second portion the object, the second chunk being generated from a plurality of depth images of the second portion of the object;

computing, by the processor, a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks;

aligning, by the processor, the first chunk with the second chunk in accordance with the registration; and

outputting, by the processor, a 3D model corresponding to the first chunk merged with the second chunk.

2. The method of claim 1, wherein the computing the registration of the first chunk with the second chunk comprises:

computing a first cuboid bounding box of the first chunk;

computing a second cuboid bounding box of the second chunk;

finding a plurality of rigid transformations of the first cuboid onto the second cuboid;

computing a plurality of registration costs, each of the registration costs corresponding to one of the rigid transformations; and

selecting a rigid transformation of the rigid transformation in accordance with the registration costs, the registration comprising the rigid transformation.

3. The method of claim 1, wherein the computing the registration of the first chunk with the second chunk comprises:

extracting, by the processor, first key points from the first chunk;

extracting, by the processor, second key points from the second chunk; and

computing a rigid transformation from the second key points to the first key points.

4. The method of claim 3, further comprising:

computing a registration cost of the rigid transformation;

modifying the registration cost in accordance with inconsistency between matching points of the first chunk and the second chunk; and

computing the registration in accordance with a rigid transformation having a minimized registration cost.

5. The method of claim 1, wherein the computing the registration of the first chunk with the second chunk comprises:

receiving a user supplied transformation of the first chunk, the user supplied transformation; and

applying the user supplied transformation to the first chunk.

6. The method of claim 5, wherein the computing the registration of the first chunk with the second chunk further comprises:

computing a first cuboid bounding box of the first chunk;

computing a second cuboid bounding box of the second chunk;

identifying three vertices of the first chunk;

transforming the three vertices of the first cuboid in accordance with the user supplied transformation;

identifying the three vertices of the second cuboid corresponding to the transformed three vertices of the first cuboid;

computing a vertex transformation of the three vertices of the first cuboid to the three corresponding vertices of the second cuboid;

initializing the transformation aligning corresponding portions of the first and second chunks in accordance with the user supplied transformation and the vertex transformation; and

refining the transformation aligning corresponding portions of the first and second chunks in accordance with a local registration method.

7. The method of claim 1, wherein the aligning the first chunk with the second chunk in accordance with the registration comprises:

transforming the first chunk in accordance with the registration;

combining matching points between the first chunk and the second chunk; and

adding unique points from the first chunk and the second chunk.

8. The method of claim 1, wherein the method further comprises:

when the registration of the first chunk with the second chunk fails;

receiving an additional depth frame corresponding to the second chunk;

updating the 3D model of the second chunk in accordance with the additional depth frame to compute an updated second chunk; and

computing the registration of the first chunk with the updated second chunk.

9. The method of claim 1, wherein the 3D model of the first portion of the object corresponds to the object in a first configuration, and the 3D model of the second portion of the object corresponds to the object in a second configuration.

10. The method of claim 1, further comprising

detecting portions of the depth images of the second portion of the object corresponding to human hands;

removing the portions of the depth images corresponding to the human hands; and

generating the second chunk from the depth images with the portions corresponding to human hands removed from the depth images.

11. A system for generating a three-dimensional (3D) scan of an object, the system comprising:

a depth camera system comprising a sensor;

a display device;

a processor coupled to depth camera system and the display device; and

memory storing instructions that, when executed by the processor, cause the processor to;

control the depth camera system to capture a first plurality of depth images;

compute a first chunk comprising a 3D model of a first portion of the object generated from the first plurality of depth images;

control the depth camera system to capture a second plurality of depth images;

compute a second chunk comprising a 3D model of a second portion of the object generated from the second plurality of depth images;

compute a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks;

align the first chunk with the second chunk in accordance with the registration; and

display, on the display device, a 3D model corresponding to the first chunk merged with the second chunk.

12. The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the registration of the first chunk with the second chunk by:

computing, by the processor, a first cuboid bounding box of the first chunk;

computing, by the processor, a second cuboid bounding box of the second chunk;

finding, by the processor, a plurality of rigid transformations of the first cuboid onto the second cuboid;

computing, by the processor, a plurality of registration costs, each of the registration costs corresponding to one of the rigid transformations; and

selecting, by the processor, a rigid transformation of the rigid transformation in accordance with the registration costs, the registration comprising the rigid transformation.

13. The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the registration of the first chunk with the second chunk by:

extracting, by the processor, first key points from the first chunk;

extracting, by the processor, second key points from the second chunk; and

computing a rigid transformation from the second key points to the first key points.

14. The system of claim 13, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the registration of the first chunk with the second chunk by:

computing a registration cost of the rigid transformation;

modifying the registration cost in accordance with inconsistency between matching points of the first chunk and the second chunk; and

computing the registration in accordance with a rigid transformation having a minimized registration cost.

15. The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

receive a user supplied transformation of the first chunk, the user supplied transformation; and

apply the user supplied transformation to the first chunk.

16. The system of claim 15, wherein the display device is a touchscreen, and

wherein the user supplied transformation is received via a touch input supplied to the touchscreen.

17. The system of claim 15, wherein the user supplied transformation is detected as a three dimensional gesture detected by the depth camera system.

18. The system of claim 15, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the registration of the first chunk with the second chunk by:

computing a first cuboid bounding box of the first chunk;

computing a second cuboid bounding box of the second chunk;

identifying three vertices of the first chunk;

transforming the three vertices of the first cuboid in accordance with the user supplied transformation;

identifying the three vertices of the second cuboid corresponding to the transformed three vertices of the first cuboid;

computing a vertex transformation of the three vertices of the first cuboid to the three corresponding vertices of the second cuboid;

initializing the transformation aligning corresponding portions of the first and second chunks in accordance with the user supplied transformation and the vertex transformation; and

refining the transformation aligning corresponding portions of the first and second chunks in accordance with a local registration method.

19. The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to align the first chunk with the second chunk by:

transforming the first chunk in accordance with the registration;

combining matching points between the first chunk and the second chunk; and

adding unique points from the first chunk and the second chunk.

20. The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to, when the registration of the first chunk with the second chunk fails:

receive an additional depth frame corresponding to the second chunk;

update the 3D model of the second chunk in accordance with the additional depth frame to compute an updated second chunk; and

compute the registration of the first chunk with the updated second chunk.

21. The system of claim 11, wherein the 3D model of the first portion of the object corresponds to the object in a first configuration, and the 3D model of the second portion of the object corresponds to the object in a second configuration.

22. The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

detect portions of the depth images of the second portion of the object corresponding to human hands;

remove the portions of the depth images corresponding to the human hands; and

generate the second chunk from the depth images with the portions corresponding to human hands removed from the depth images.

23. The system of claim 11, wherein the display device is integrated into a pair of augmented reality glasses.

24. The system of claim 23, wherein the depth camera system is integrated into the pair of augmented reality glasses.

25. A method for assembling a plurality of chunks corresponding to separate three-dimensional (3D) models of different portions of an object, the chunks being displayed in a graphical user interface, the method comprising:

receiving, by a processor, a selection of a first chunk of the plurality of chunks, the first chunk comprising a 3D model of a first portion of the object;

receiving, by the processor, a selection of a second chunk of the plurality of chunks, the second chunk comprising a 3D model of a second portion of the object;

computing, by the processor, a registration of the first chunk with the second chunk, the registration corresponding to a transformation aligning corresponding portions of the first and second chunks;

aligning, by the processor, the first chunk with the second chunk in accordance with the registration; and

displaying, by the processor, the first chunk assembled with the second chunk as a chunk assembly.

26. The method of claim 25, further comprising:

receiving user input to transform a view of the chunk assembly, the user input comprising at least one transformation of the group comprising;

a rotation;

a translation; and

a zoom.

27. The method of claim 25, further comprising:

receiving a user input to transform the first chunk, the user input comprising at least one transformation of the group comprising a rotation and a translation; and

updating the displaying of the first chunk in accordance with the user input.

28. The method of claim 25, further comprising displaying a confidence of the chunk assembly.

29. The method of claim 25, wherein points in the chunk assembly are colored in accordance with whether the points originated from the first chunk or the second chunk.

30. The method of claim 25, wherein the plurality of chunks are associated with a scanning project, the scanning project comprising one or more versions of assemblies of one or more chunks, each version being associated with metadata storing a collection of chunks that are included an assembly associated with the version.

31. The method of claim 25, wherein at least one chunk of the plurality of chunks is received from a remote source.