Automated validation of condition of assembly

• US 10,445,873 B2
• Filed: 02/23/2017
• Issued: 10/15/2019
• Est. Priority Date: 02/23/2017
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Szarski, Martin, Haeusler, Phillip, Bain, David Michael, Bain, Richard, Glynn, Andrew K., Steele, Peter Nathan
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US15/441,082
• Publication Number:
  US 20180240226A1
• Time in Patent Office: 964 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30136 Metal
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/001 using an image reference approach

Abstract

A method and apparatus for performing an automated validation of a condition of assembly for a structure. A plurality of images of the structure are registered to a computer model of the structure in which an image in the plurality of images captures a portion of the structure. Each image in the plurality of images is segmented based on registration of the plurality of images to the computer model to form a plurality of image sections. A final score is generated for the condition of assembly of the structure based on whether each image section in the plurality of image sections meets a corresponding condition in which the final score indicates whether the condition of assembly is valid.

Claims

1. A method of performing an automated validation of a condition of assembly for a structure, the method comprising:

registering a plurality of images of the structure to a computer model of the structure in which an image in the plurality of images captures a portion of the structure;

segmenting each image in the plurality of images based on registration of the plurality of images to the computer model to form a plurality of image sections such that each image section of the plurality of image sections is expected, based on registration of the plurality of images to the computer model, to capture a corresponding feature of interest from features of the structure identified in the computer model; and

generating a final score for the condition of assembly of the structure based on whether each image section in the plurality of image sections meets a corresponding condition in which the final score indicates whether the condition of assembly is valid.

2. The method of claim 1 further comprising:

generating image data for use in generating the plurality of images of the structure using a sensor system, wherein the sensor system comprises a plurality of sensors positioned at a plurality of fixed positions relative to the structure.

3. The method of claim 1 further comprising:

generating the plurality of images of the structure using a sensor system coupled to an automated guided vehicle.

4. The method of claim 3, wherein generating the image data comprises:

moving the automated guided vehicle along a predetermined path relative to the structure to allow the sensor system to be moved into a set of positions relative to the structure; and

generating an image at each position of the set of positions using the sensor system.

5. The method of claim 1,

wherein the corresponding feature of interest is a single feature of interest selected from one of a hole, a fastener installed in the hole, an absence of the hole, or an absence of the fastener in the hole.

6. The method of claim 1, wherein generating the final score comprises:

computing a percentage of the plurality of image sections that meet the corresponding condition for each respective image section in the plurality of image sections, wherein the corresponding condition is selected from one of a presence of a hole, a presence of a fastener installed in the hole, an absence of a hole, and an absence of the fastener in the hole.

7. The method of claim 1, wherein generating the final score comprises:

comparing an image section in the plurality of image sections to a corresponding portion of the computer model of the structure to determine whether the image section meets the corresponding condition, wherein the corresponding condition is selected from one of a presence of a hole, a presence of a fastener installed in the hole, an absence of a hole, and an absence of the fastener in the hole.

8. The method of claim 1, wherein generating the final score comprises:

computing an initial score for the condition of assembly of the structure based on whether each image section in the plurality of image sections meets the corresponding condition;

determining whether or not any invalidating events have been identified; and

adjusting the initial score to create the final score in response to a determination that at least one invalidating event has been identified.

9. The method of claim 8, wherein adjusting the initial score comprises:

reducing the initial score to zero percent in response to the determination that at least one invalidating event has been identified.

10. The method of claim 8, wherein adjusting the initial score comprises:

making an adjustment to the initial score for each invalidating event identified, wherein the adjustment is based on an importance of each invalidating event to the condition of assembly for the structure.

11. The method of claim 1 further comprising:

displaying the final score on a graphical user interface on a display system.

12. The method of claim 1 further comprising:

sending a report that includes the final score to a computerized numerically controlled machine for processing.

13. The method of claim 12 further comprising:

adjusting programming for the computerized numerically controlled machine based on the report.

14. The method of claim 1 further comprising:

generating the plurality of images of the structure using a sensor system, wherein the structure is selected from one of an aircraft, a fuselage structure, a wing structure, a tail structure, and a nose structure.

15. An apparatus comprising:

a sensor system that generates image data for use in generating a plurality of images of a structure in which an image in the plurality of images captures a portion of the structure;

a processor in communication with the sensor system, wherein the processor comprises;

an image processing component that receives the plurality of images;

registers the plurality of images to a computer model of the structure; and

segments each image in the plurality of images based on registration of the plurality of images to the computer model to form a plurality of image sections such that each image section of the plurality of image sections is expected, based on registration of the plurality of images to the computer model, to capture a corresponding feature of interest from features of the structure identified in the computer model; and

a comparison component that generates a final score for the condition of assembly for the structure based on whether each image section in the plurality of image sections meets a corresponding condition in which the final score indicates whether the condition of assembly is valid.

16. The apparatus of claim 15, wherein the sensor system comprises a plurality of sensors positioned in a plurality of fixed positions relative to the structure.

17. The apparatus of claim 15, wherein the sensor system is an imaging system coupled to an automated guided vehicle.

18. The apparatus of claim 17 further comprising:

the automated guided vehicle, wherein the imaging system is at least one of removably attached to the automated guided vehicle, permanently affixed to the automated guided vehicle, or built-in as part of the automated guided vehicle.

19. The apparatus of claim 17, wherein the automated guided vehicle is moved to a set of positions along a path relative to the structure and wherein the imaging system is used to generate at least one image of the plurality of images of the structure at each of the set of positions along the path.

20. The apparatus of claim 19, wherein the path is a predetermined path comprising an optimal set of positions that allow an entirety of an area of the structure that is of interest to be captured using the fewest number of images.

21. The apparatus of claim 15, wherein segmenting each image section in the plurality of image sections for use in generating the final score reduces time and processing resources needed to perform automated validation of the condition of assembly of the structure.

22. The apparatus of claim 15, wherein the corresponding feature of interest is a single feature of interest selected from one of a hole, a fastener installed in the hole, an absence of the hole, or an absence of the fastener in the hole.

23. The apparatus of claim 15, wherein each image section in the plurality of image sections captures a portion of the structure that is expected to have at least one feature of interest.

24. The apparatus of claim 15, wherein the corresponding condition for an image section in the plurality of image sections is selected from one of a presence of a hole, a presence of a fastener installed in the hole, an absence of the hole, and an absence of the fastener in the hole and wherein the final score is computed based on a percentage of the plurality of images sections that meet the corresponding condition.

25. The apparatus of claim 15 further comprising:

a display system in communication with the processor, wherein the final score is displayed on a graphical user interface on the display system.

26. The apparatus of claim 15, wherein the processor generates a report that includes the final score and sends the report to a computerized numerically controlled machine for processing.

27. A method for performing an automated validation process to validate a condition of assembly of an aircraft structure, the method comprising:

generating image data for use in generating the plurality of images of the aircraft structure using a sensor system attached to an automated guided vehicle, wherein an image in the plurality of images captures a portion of the aircraft structure;

registering the plurality of images to a computer model of the aircraft structure;

segmenting each image in the plurality of images based on registration of the plurality of images to the computer model to form a plurality of image sections such that each image section of the plurality of image sections is expected, based on registration of the plurality of images to the computer model, to capture a corresponding feature of interest from features of the aircraft structure identified in the computer model;

generating a final score for the aircraft structure based on whether each image section in the plurality of image sections meets a corresponding condition with respect to the computer model of the aircraft structure in which the final score indicates whether the condition of assembly for the structure is valid; and

displaying the final score in a graphical user interface on a display system.

28. The method of claim 27, wherein the aircraft structure is a fuselage structure.

29. The method of claim 27 further comprising:

sending a report that includes the final score to a computerized numerically controlled machine, wherein the computerized numerically controlled machine initiates a next stage of assembly when the final score indicates that the condition of assembly is valid.

30. A method comprising:

autonomously registering a plurality of images of the structure to a computer model of the structure in which an image in the plurality of images captures a portion of the structure; and

autonomously segmenting each image in the plurality of images based on registration of the plurality of images to the computer model to form a plurality of image sections such that each image section of the plurality of image sections is expected, based on registration of the plurality of images to the computer model, to capture a corresponding feature of interest from features of the structure identified in the computer model.

31. The method of claim 30 further comprising:

autonomously moving an automated guided vehicle along a path relative to the structure to generate the plurality of images.

32. The method of claim 30 further comprising:

autonomously determining whether a condition of assembly for the structure is valid using the plurality of image sections and the computer model of the structure.