SYSTEMS AND METHODS FOR DEFECT DETECTION

• US 20180211373A1
• Filed: 01/09/2018
• Published: 07/26/2018
• Est. Priority Date: 01/20/2017
• Status: Abandoned Application

• Current Assignee: Aquifi, Inc.
• Original Assignee: Aquifi, Inc.
• Inventors: Stoppa, Michele, Peruch, Francesco, Pasqualotto, Giuliano, Hazeghi, Aryan, Salvagnini, Pietro, Dal Mutto, Carlo, Trachewsky, Jason, Tieu, Kinh
• Application Number:
  US15/866,217
• Publication Number:
  US 20180211373A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

A method for detecting a defect in an object includes: capturing, by one or more depth cameras, a plurality of partial point clouds of the object from a plurality of different poses with respect to the object; merging, by a processor, the partial point clouds to generate a merged point cloud; computing, by the processor, a three-dimensional (3D) multi-view model of the object; detecting, by the processor, one or more defects of the object in the 3D multi-view model; and outputting, by the processor, an indication of the one or more defects of the object.

Claims

1. A method for detecting a defect in an object, the method comprising:

capturing, by one or more depth cameras, a plurality of partial point clouds of the object from a plurality of different poses with respect to the object;

merging, by a processor, the partial point clouds to generate a merged point cloud;

computing, by the processor, a three-dimensional (3D) multi-view model of the object;

detecting, by the processor, one or more defects of the object in the 3D multi-view model; and

outputting, by the processor, an indication of the one or more defects of the object.

2. The method of claim 1, wherein the detecting the one or more defects comprises:

aligning the 3D multi-view model with a reference model;

comparing the 3D multi-view model to the reference model to compute a plurality of differences between corresponding regions of the 3D multi-view model and the reference model; and

detecting the one or more defects in the object when one or more of the plurality of differences exceeds a threshold.

3. The method of claim 2, wherein the comparing the 3D multi-view model to the reference model comprises:

dividing the 3D multi-view model into a plurality of regions;

identifying corresponding regions of the reference model;

detecting locations of features in the regions of the 3D multi-view model;

computing distances between detected features in the regions of the 3D multi-view model and locations of features in the corresponding regions of the reference model; and

outputting the distances as the plurality of differences.

4. The method of claim 1, further comprising:

computing a plurality of features based on the 3D multi-view model, the features comprising color, texture, and shape; and

assigning a classification to the object in accordance with the plurality of features, the classification comprising one of;

one or more classifications, each classification corresponding to a different type of defect; and

a clean classification.

5. The method of claim 4 wherein the computing the plurality of features comprises:

rendering one or more two-dimensional views of the 3D multi-view model; and

computing the plurality of features based on the one or more two-dimensional views of the object.

6. The method of claim 4, wherein the computing the plurality of features comprises:

dividing the 3D multi-view model into a plurality of regions; and

computing the plurality of features based on the plurality of regions of the 3D multi-view model.

7. The method of claim 4, wherein the assigning the classification to the object in accordance with the plurality of features is performed by a convolutional neural network, and

wherein the convolutional neural network is trained by:

receiving a plurality of training 3D models of objects and corresponding training classifications;

computing a plurality of feature vectors from the training 3D models by the convolutional neural network;

computing parameters of the convolutional neural network;

computing a training error metric between the training classifications of the training 3D models with outputs of the convolutional neural network configured based on the parameters;

computing a validation error metric in accordance with a plurality of validation 3D models separate from the training 3D models;

in response to determining that the training error metric and the validation error metric fail to satisfy a threshold, generating additional 3D models with different defects to generate additional training data;

in response to determining that the training error metric and the validation error metric satisfy the threshold, configuring the neural network in accordance with the parameters;

receiving a plurality of test 3D models of objects with unknown classifications; and

classifying the test 3D models using the configured convolutional neural network.

8. The method of claim 4, wherein the assigning the classification to the object in accordance with the plurality of features is performed by:

comparing each of the features to a corresponding previously observed distribution of values of the feature;

assigning the clean classification in response to determining that all of the values of the features are within a typical range; and

assigning a defect classification for each feature of the plurality of features that are in outlier portions of the corresponding previously observed distribution.

9. The method of claim 1, further comprising displaying the indication of the one or more defects on a display device.

10. The method of claim 9, wherein the display device is configured to display the 3D multi-view model, and

wherein the one or more defects are displayed as a heat map overlaid on the 3D multi-view model.

11. The method of claim 1, wherein the indication of the one or more defects of the object controls movement of the object out of a normal processing route.

12. The method of claim 1, wherein the object is located on a conveyor system, and

wherein the one or more depth cameras are arranged around the conveyor system to image the object as the object moves along the conveyor system.

13. The method of claim 12, wherein the point clouds are captured at different times as the object moves along conveyor system.

14. The method of claim 1, wherein the 3D multi-view model comprises a 3D mesh model.

15. The method of claim 1, wherein the 3D multi-view model comprises a 3D point cloud.

16. The method of claim 1, wherein the 3D multi-view model comprises a plurality of two-dimensional images.

17. A system for detecting a defect in an object, the system comprising:

a plurality of depth cameras arranged to have a plurality of different poses with respect to the object;

a processor in communication with the depth cameras; and

a memory storing instructions that, when executed by the processor, cause the processor to.

receive, from the one or more depth cameras, a plurality of partial point clouds of the object from the plurality of different poses with respect to the object;

merge the partial point clouds to generate a merged point cloud;

compute a three-dimensional (3D) multi-view model of the object;

detect one or more defects of the object in the 3D multi-view model; and

output an indication of the one or more defects of the object.

18. The system of claim 17, wherein the memory further stores instructions that, when executed by the processor, cause the processor to detect the one or more defects by:

aligning the 3D multi-view model with a reference model;

comparing the 3D multi-view model to the reference model to compute a plurality of differences between corresponding regions of the 3D multi-view model and the reference model; and

detecting the one or more defects in the object when one or more of the plurality of differences exceeds a threshold.

19. The system of claim 18, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compare the 3D multi-view model to the reference model by:

dividing the 3D multi-view model into a plurality of regions;

identifying corresponding regions of the reference model;

detecting locations of features in the regions of the 3D multi-view model;

computing distances between detected features in the regions of the 3D multi-view model and locations of features in the corresponding regions of the reference model; and

outputting the distances as the plurality of differences.

20. The system of claim 17, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

compute a plurality of features based on the 3D multi-view model, the features comprising color, texture, and shape; and

assign a classification to the object in accordance with the plurality of features, the classification comprising one of;

one or more classifications, each classification corresponding to a different type of defect; and

a clean classification.

21. The system of claim 20, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

render one or more two-dimensional views of the 3D multi-view model; and

compute the plurality of features based on the one or more two-dimensional views of the object.

22. The system of claim 20, wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the plurality of features by:

dividing the 3D multi-view model into a plurality of regions; and

computing the plurality of features based on the plurality of regions of the 3D multi-view model.

23. The system of claim 20, wherein the memory further stores instructions that, when executed by the processor, cause the processor to assign the classification to the object using a convolutional neural network, and

wherein the convolutional neural network is trained by:

receiving a plurality of training 3D models of objects and corresponding training classifications;

computing a plurality of feature vectors from the training 3D models by the convolutional neural network;

computing parameters of the convolutional neural network;

computing a training error metric between the training classifications of the training 3D models with outputs of the convolutional neural network configured based on the parameters;

computing a validation error metric in accordance with a plurality of validation 3D models separate from the training 3D models;

in response to determining that the training error metric and the validation error metric fail to satisfy a threshold, generating additional 3D models with different defects to generate additional training data;

in response to determining that the training error metric and the validation error metric satisfy the threshold, configuring the neural network in accordance with the parameters;

receiving a plurality of test 3D models of objects with unknown classifications; and

classifying the test 3D models using the configured convolutional neural network.

24. The system of claim 20, wherein the memory further stores instructions that, when executed by the processor, cause the processor to assign the classification to the object in accordance with the plurality of features by:

comparing each of the features to a corresponding previously observed distribution of values of the feature;

assigning the clean classification in response to determining that all of the values of the features are within a typical range; and

assigning a defect classification for each feature of the plurality of features that are in outlier portions of the corresponding previously observed distribution.

25. The system of claim 17, further comprising a display device,

wherein the memory further stores instructions that, when executed by the processor, cause the processor to display the indication of the one or more defects on the display device.

26. The system of claim 25, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

display, on the display device, the indication of the one or more defects as a heat map overlaid on the 3D multi-view model.

27. The system of claim 17, wherein the memory further stores instructions that, when executed by the processor, cause the processor to control the movement of the object out of a normal processing route based on the indication of the one or more defects.

28. The system of claim 17, further comprising a conveyor system,

wherein the object is moving on the conveyor system, and

wherein the one or more depth cameras are arranged around the conveyor system to image the object as the object moves along the conveyor system.

29. The system of claim 28, wherein the point clouds are captured at different times as the object moves along the conveyor system.

30. The system of claim 17, wherein the 3D multi-view model comprises a 3D mesh model.

31. The system of claim 17, wherein the 3D multi-view model comprises a 3D point cloud.

32. The system of claim 17, wherein the 3D multi-view model comprises a plurality of two-dimensional images.