SYSTEMS AND METHODS FOR VISUAL INSPECTION BASED ON AUGMENTED REALITY

• US 20190096135A1
• Filed: 09/26/2018
• Published: 03/28/2019
• Est. Priority Date: 09/26/2017
• Status: Abandoned Application

• Current Assignee: Aquifi, Inc.
• Original Assignee: Aquifi, Inc.
• Inventors: Dal Mutto, Carlo, Trachewsky, Jason, Zuccarino, Tony
• Application Number:
  US16/143,400
• Publication Number:
  US 20190096135A1
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10008 from scanner, fax or copier
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/0006 : using a design-rule based approach
• CPC Class Codes: G06T 7/001 : using an image reference approach
• CPC Class Codes: G06T 7/11 : Region-based segmentation
• CPC Class Codes: G06T 7/50 : Depth or shape recovery
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 2201/10 : Recognition assisted with metadata

Abstract

A system for visual inspection includes: a scanning system configured to capture images of an object and to compute a three-dimensional (3-D) model of the object based on the captured images; an inspection system configured to: compute a descriptor of the object based on the 3-D model of the object; retrieve metadata corresponding to the object based on the descriptor; and compute a plurality of inspection results based on the retrieved metadata and the 3-D model of the object; and a display device system including: a display; a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: generate overlay data from the inspection results; and show the overlay data on the display, the overlay data being aligned with a view of the object through the display.

Claims

1. A system for visual inspection, comprising:

a scanning system configured to capture images of an object and to compute a three-dimensional (3-D) model of the object based on the captured images;

an inspection system configured to;

compute a descriptor of the object based on the 3-D model of the object;

retrieve metadata corresponding to the object based on the descriptor; and

compute a plurality of inspection results based on the retrieved metadata and the 3-D model of the object; and

a display device system comprising;

a display;

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to;

generate overlay data from the inspection results; and

show the overlay data on the display, the overlay data being aligned with a view of the object through the display.

2. The system of claim 1, wherein the display device system comprises an augmented reality head-mounted device (AR-HMD) comprising the display, and

wherein the display is transparent to provide the view of the object.

3. The system of claim 2, wherein the AR-HMD comprises one or more sensing components configured to capture information about an environment near the AR-HMD and an orientation of the AR-HMD, and

wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

compute a relative pose of the object with respect to the view of the object through the display of the AR-HMD based on the information from the sensing components; and

transform a position of the overlay data in the display in accordance with the relative pose of the object with respect to the view of the object through the display of the AR-HMD.

4. The system of claim 3, wherein the one or more sensing components comprise a depth camera.

5. The system of claim 3, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

detect a change in the relative pose of the object with respect to the view of the object through the display of the AR-HMD based on the information from the sensing components; and

transform the position of the overlay data in accordance with the change in the relative pose of the object with respect to the view of the object through the display of the AR-HMD.

6. The system of claim 3, wherein the display of the AR-HMD comprises a left lens and a right lens, and

wherein the instructions to transform the position of the overlay data comprise instructions that cause the processor to:

compute a first position of the overlay data in accordance with the relative pose of the object with respect to a first view of the object through the left lens of the display of the AR-HMD; and

compute a second position of the overlay data in accordance with the relative pose of the object with respect to a second view of the object through the right lens of the display of the AR-HMD.

7. The system of claim 1, wherein the display device system comprises a camera,

wherein the display comprises a display panel, and

wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

control the camera to capture images of the object; and

show the captured images of the object on the display to provide the view of the object.

8. The system of claim 7, wherein the display device system comprises one or more sensing components configured to capture information about an environment near the camera, and

wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

compute a relative pose of the object with respect to the view of the object through the display of the display device system based on the information from the sensing components; and

transform a position of the overlay data in the display in accordance with the relative pose of the object with respect to the view of the object through the display of the display device system.

9. The system of claim 8, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

detect a change in the relative pose of the object with respect to the view of the object through the display of the display device system based on the information from the sensing components; and

transform the position of the overlay data in accordance with the change in the relative pose of the object with respect to the view of the object through the display of the display device system.

10. The system of claim 8, wherein the one or more sensing components comprise the camera.

11. The system of claim 1, wherein the plurality of inspection results comprises defects detected by the inspection system.

12. The system of claim 11, wherein the inspection system is configured to detect defects by:

retrieving, from the metadata, one or more expected measurement values of the object;

measuring one or more measurement values of the 3-D model of the object;

comparing the one or more measurement values with corresponding ones of the one or more expected measurement values; and

detecting a defect when a measurement value of the one or more measurement values differs from a corresponding one of the one or more expected measurement values.

13. The system of claim 11, wherein the inspection system is configured to detect defects by:

retrieving, from the metadata, a reference 3-D model of a canonical instance of a class corresponding to the object;

aligning the 3-D model of the object with the reference 3-D model;

comparing the 3-D model of the object to the reference 3-D model to compute a plurality of differences between corresponding regions of the 3-D model of the object and the reference 3-D model; and

detecting one or more defects in the object when one or more of the plurality of differences exceeds a threshold.

14. The system of claim 13, wherein the comparing the 3-D model of the object to the reference 3-D model comprises:

dividing the 3-D model of the object into a plurality of regions;

identifying corresponding regions of the reference 3-D model;

detecting locations of features in the regions of the 3-D model of the object;

computing distances between detected features in the regions of the 3-D model of the object and locations of features in the corresponding regions of the reference 3-D model; and

outputting the distances as the plurality of differences.

15. The system of claim 13, wherein the reference 3-D model is computed by:

scanning a plurality of training objects corresponding to the class to generate a plurality of training 3-D models;

removing outliers from the plurality of training 3-D models to generate a plurality of typical training 3-D models; and

computing an average of the plurality of typical training 3-D models to generate the reference 3-D model.

16. The system of claim 11, wherein the inspection system is configured to detect defects by:

retrieving, from the metadata, a convolutional stage of a convolutional neural network and a defect detector;

rendering one or more views of the 3-D model of the object;

computing a descriptor by supplying the one or more views of the 3-D model of the object to the convolutional stage of the convolutional neural network;

supplying the descriptor to the defect detector to compute one or more defect classifications of the object; and

outputting the one or more defect classifications of the object.

17. The system of claim 11, wherein the inspection system is configured to detect defects by:

retrieving, from the metadata, one or more rules, each of the rules comprising an explicitly defined detector;

rendering one or more views of the 3-D model of the object; and

applying the explicitly defined detector of each of the one or more rules to the one or more views of the 3-D model of the object to compute one or more defect classifications of the object.

18. The system of claim 11, wherein the inspection system is configured to detect defects by:

retrieving, from the metadata, a generative model trained based on a plurality of training 3-D models of a representative sample of non-defective objects; and

supplying the 3-D model of the object to the generative model to compute one or more defect classifications of the object.

19. The system of claim 11, wherein the defects comprise location-specific defects,

wherein the inspection results comprise a result 3-D model, the result 3-D model identifying a location of at least one defect of the object, and

wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

align the result 3-D model with the view of the object through the display; and

show the result 3-D model overlaid on the view of the object through the display.

20. The system of claim 19, wherein the result 3-D model indicates a magnitude of the at least one defect at the location, and

wherein the magnitude is shown as a color overlay on the view of the object through the display.

21. The system of claim 1, wherein the memory further stores instructions that, when executed by the processor, cause the processor to show non-location specific information of the inspection results in association with the view of the object through the display.

22. A method for visual inspection, comprising:

capturing a plurality of images of an object using a scanning system;

computing a three-dimensional (3-D) model of the object based on the plurality of images;

computing, by an inspection system comprising a processor and memory, a descriptor of the object based on the 3-D model of the object;

retrieving, by the inspection system, metadata corresponding to the object based on the descriptor;

computing, by the inspection system, a plurality of inspection results based on the retrieved metadata and the 3-D model of the object;

generating overlay data from the inspection results; and

showing the overlay data on a display of a display device system, the overlay data being aligned with a view of the object through the display.

23. The method of claim 22, wherein the display device system comprises an augmented reality head-mounted device (AR-HMD) comprising the display, and

wherein the display is transparent to provide the view of the object.

24. The method of claim 23, wherein the AR-HMD comprises one or more sensing components configured to capture information about an environment near the AR-HMD and an orientation of the AR-HMD, and

wherein the method further comprises:

computing, by the display device system, a relative pose of the object with respect to the view of the object through the display of the AR-HMD based on the information from the sensing components; and

transforming a position of the overlay data in the display in accordance with the relative pose of the object with respect to the view of the object through the display of the AR-HMD.

25. The method of claim 24, wherein the one or more sensing components comprise a depth camera.

26. The method of claim 24, further comprising:

detecting a change in the relative pose of the object with respect to the view of the object through the display of the AR-HMD based on the information from the sensing components; and

transforming the position of the overlay data in accordance with the change in the relative pose of the object with respect to the view of the object through the display of the AR-HMD.

27. The method of claim 24, wherein the display of the AR-HMD comprises a left lens and a right lens, and

wherein the transforming the position of the overlay data comprises:

computing a first position of the overlay data in accordance with the relative pose of the object with respect to a first view of the object through the left lens of the display of the AR-HMD; and

computing a second position of the overlay data in accordance with the relative pose of the object with respect to a second view of the object through the right lens of the display of the AR-HMD.

28. The method of claim 22, wherein the display device system comprises a camera,

wherein the display comprises a display panel, and

wherein the method further comprises:

controlling the camera to capture images of the object; and

showing the captured images of the object on the display to provide the view of the object.

29. The method of claim 28, wherein the display device system comprises one or more sensing components configured to capture information about an environment near the camera, and

wherein the method further comprises:

computing a relative pose of the object with respect to the view of the object through the display of the display device system based on the information from the sensing components; and

transforming a position of the overlay data in the display in accordance with the relative pose of the object with respect to the view of the object through the display of the display device system.

30. The method of claim 29, further comprising:

detecting a change in the relative pose of the object with respect to the view of the object through the display of the display device system based on the information from the sensing components; and

transforming the position of the overlay data in accordance with the change in the relative pose of the object with respect to the view of the object through the display of the display device system.

31. The method of claim 29, wherein the one or more sensing components comprise the camera.

32. The method of claim 22, wherein the plurality of inspection results comprises defects detected by the inspection system.

33. The method of claim 32, wherein the inspection system detects defects by:

retrieving, from the metadata, one or more expected measurement values of the object;

measuring one or more measurement values of the 3-D model of the object;

comparing the one or more measurement values with corresponding ones of the one or more expected measurement values; and

detecting a defect when a measurement value of the one or more measurement values differs from a corresponding one of the one or more expected measurement values.

34. The method of claim 32, wherein the inspection system detects defects by:

retrieving, from the metadata, a reference 3-D model of a canonical instance of a class corresponding to the object;

aligning the 3-D model of the object with the reference 3-D model;

comparing the 3-D model of the object to the reference 3-D model to compute a plurality of differences between corresponding regions of the 3-D model of the object and the reference 3-D model; and

detecting one or more defects in the object when one or more of the plurality of differences exceeds a threshold.

35. The method of claim 34, wherein the comparing the 3-D model of the object to the reference 3-D model comprises:

dividing the 3-D model of the object into a plurality of regions;

identifying corresponding regions of the reference 3-D model;

detecting locations of features in the regions of the 3-D model of the object;

computing distances between detected features in the regions of the 3-D model of the object and locations of features in the corresponding regions of the reference 3-D model; and

outputting the distances as the plurality of differences.

36. The method of claim 34, wherein the reference 3-D model is computed by:

scanning a plurality of training objects corresponding to the class to generate a plurality of training 3-D models;

removing outliers from the plurality of training 3-D models to generate a plurality of typical training 3-D models; and

computing an average of the plurality of typical training 3-D models to generate the reference 3-D model.

37. The method of claim 32, wherein the inspection system detects defects by:

retrieving, from the metadata, a convolutional stage of a convolutional neural network and a defect detector;

rendering one or more views of the 3-D model of the object;

computing a descriptor by supplying the one or more views of the 3-D model of the object to the convolutional stage of the convolutional neural network;

supplying the descriptor to the defect detector to compute one or more defect classifications of the object; and

outputting the one or more defect classifications of the object.

38. The method of claim 32, wherein the inspection system detects defects by:

retrieving, from the metadata, one or more rules, each of the rules comprising an explicitly defined detector;

rendering one or more views of the 3-D model of the object; and

applying the explicitly defined detector of each of the one or more rules to the one or more views of the 3-D model of the object to compute one or more defect classifications of the object.

39. The method of claim 32, wherein the inspection system detects defects by:

retrieving, from the metadata, a generative model trained based on a plurality of training 3-D models of a representative sample of non-defective objects; and

supplying the 3-D model of the object to the generative model to compute one or more defect classifications of the object.

40. The method of claim 32, wherein the defects comprise location-specific defects,

wherein the inspection results comprise a result 3-D model, the result 3-D model identifying a location of at least one defect of the object, and

wherein the method further comprises:

aligning the result 3-D model with the view of the object through the display; and

showing the result 3-D model overlaid on the view of the object through the display.

41. The method of claim 40, wherein the result 3-D model indicates a magnitude of the at least one defect at the location, and

wherein the magnitude is shown as a color overlay on the view of the object through the display.

42. The method of claim 22, further comprising showing non-location specific information of the inspection results in association with the view of the object through the display.