Portable augmented reality

• US 10,026,227 B2
• Filed: 12/01/2014
• Issued: 07/17/2018
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Laughlin, Brian Dale, Murrish, Richard Eugene, Vandenbrande, Jan H.
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Lhymn, Sarah
• Application Number:
  US14/556,865
• Publication Number:
  US 20150084989A1
• Time in Patent Office: 1,324 Days
• Field of Search: 345619, 345629, 345630, 345631, 345632, 345633, 345634, 345635, 345636, 345648
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038

Abstract

The different illustrative embodiments provide a method for augmenting reality that may be applied to repairs performed on a composite structure. An image may be recorded of a first layer of a composite component. The image may relate to a first repair performed at a first time. Physical data may be captured for the composite component from a surface layer of the composite component using a number of portable devices. A multi-dimensional representation of a combined augmented reality of the composite component including a display of various layers of the composite component selected by a user may be generated. A user selection may include a time restriction and a spatial restriction. The number of portable devices may display the multi-dimensional representation comprising physical data for an image of the surface of the composite component and digital data for an overlay of composite layers beyond the surface.

Claims

1. A method for generating an augmented reality presentation of a composite component comprising an aircraft skin and a patch on the aircraft skin, the method comprising:

recording an image of a first layer of a composite component, the composite component comprising a plurality of layers, wherein the first layer is one of the plurality of layers;

capturing physical data for the composite component from a surface layer of the composite component using a number of portable devices;

transmitting the physical data captured by the number of portable devices to a processor unit remote from the number of portable devices for alignment, the physical data including at least one distance from the composite component to a at least one physical object that exists in relation to the composite component;

performing a metrology process on the physical data to generate a spatial framework for composite component and the at least one physical object,

wherein the spatial framework comprises a point cloud defining a set of vertices representative of a first external surface of the composite component and a second external surface of the at least one physical object,

wherein the metrology process produces a digital definition of the composite component;

generating an affine transformation using relative alignments of the spatial framework of the composite component and the at least one physical object;

aligning, using the affine transformation, the point cloud with the digital definition of the composite component and the at least one physical object, wherein aligning includes rotation and translation to make the point cloud substantially coincident with the digital definition,

wherein substantially coincident is defined as a result where a given subset of points from the point cloud are within a predetermined distance from the digital definition;

retrieving information related to the composite component including the first layer of the composite component,

wherein the information comprises past repairs performed on the first layer of the composite component;

generating in the processor unit a multi-dimensional representation of a combined augmented reality of the composite component including a display of the first layer of the composite component; and

displaying in the number of portable devices the multi-dimensional representation comprising physical data for an image of the surface of the composite component and digital data for an overlay of the first layer of the composite component beyond the surface of the composite component,

wherein the overlay comprises inspection data related to inspection of the composite component after installation of the patch.

2. The method of claim 1, wherein the multi-dimensional representation is augmented reality.

3. The method of claim 1, wherein the point cloud further comprises a tessellated mesh that connects neighboring points in the point cloud with vector elements and wherein planar faces fill spaces between the vector elements.

4. The method of claim 1, wherein the inspection data that is overlaid comprises data entries received from a human user that indicate whether the composite component passed or failed physical inspection, as well as written notes regarding the inspection.

5. The method of claim 4, wherein the inspection data that is overlaid further comprises data automatically generated from performing an inspection of the composite component using a sensor.

6. The method of claim 5, wherein the inspection data that is overlaid further comprises notes regarding supervisory inspection of a rework of the composite component.

7. The method of claim 6, wherein the inspection data that is overlaid further comprises an environment surrounding rework of the composite component, the environment comprising all of a date, a time, weather conditions, temperature, and atmospheric conditions at a time of the rework.

8. The method of claim 7, wherein the inspection data that is overlaid comprises a list of tools and instructions for operating the tools to perform the rework.

9. The method of claim 1, wherein the inspection data that is overlaid further indicates whether the composite component is airworthy.

10. A method for displaying an augmented reality presentation of repairs to a composite component comprising an aircraft skin and a patch on the aircraft skin, the method comprising:

recording at a first time a first set of images comprising data of a first number of layers of a composite component, the composite component comprising a plurality of layers;

recording at a second time a second set of images comprising data of a second number of layers of the composite component, the first set of images and the second set of images including past repairs to the plurality of layers;

capturing physical data for a surface of the composite component using a number of portable devices, the physical data including at least one distance from the composite component to a at least one physical object that exists in relation to the composite component;

generating a number of digital definitions in a metrology process in a processor unit using the physical data,

the number of digital definitions including one or more derived geometries of sufficient quality for alignment with a digital definition of the composite component,

the metrology process generating a spatial framework for the composite component and the at least one physical object,

wherein the spatial framework comprises a point cloud defining a set of vertices representative of a first external surface of the composite component and a second external surface of the at least one physical object,

wherein the metrology process produces the digital definition of the composite component;

generating an affine transformation using relative alignments of the spatial framework of the composite component and the at least one physical object;

aligning, using the affine transformation, the point cloud with the digital definition of the composite component and the at least one physical object,

wherein aligning includes rotation and translation to make the point cloud substantially coincident with the digital definition,

wherein substantially coincident is defined as a result where a given subset of points from the point cloud are within a predetermined distance from the digital definition;

retrieving information related to the composite component including the plurality of layers;

displaying in the number of portable devices a multi-dimensional representation comprising physical data for an image of the surface of the composite component and digital data for a number of overlays of a composite layer beyond the surface of the composite component,

the number of overlays corresponding to the first set of images and the second set of images,

wherein the number of overlays comprise inspection data related to inspection of the composite component after installation of the patch; and

selecting a time and a set of images for break out display from the number of overlays.

11. The method of claim 10 further comprising selecting a time restriction for the digital data.

12. The method of claim 11, wherein the time restriction comprises a time since a last performed inspection of the composite component.

13. The method of claim 10, wherein the first number of layers comprises a first repair performed at a first time and the second number of layers comprises a second repair performed at a second time, and further comprising selecting a spatial restriction for the digital data.

14. The method of claim 13, wherein the composite component comprises and aircraft skin, and wherein the spatial restriction comprises a distance from a structure associated with the aircraft skin.

15. The method of claim 10 further comprising selecting a time restriction for the digital data and selecting a spatial restriction for the digital data.

16. A method for displaying an augmented reality presentation of a composite component comprising an aircraft skin and a patch on the aircraft skin, the method comprising:

capturing physical data for a composite component using a number of portable devices, wherein the physical data comprises past repair data, the physical data including at least one distance from the composite component to a at least one physical object that exists in relation to the composite component;

transmitting the physical data captured by the number of portable devices to a processor unit remote from the number of portable devices for alignment;

generating a number of digital definitions in a metrology process in the processor unit using the physical data, the number of digital definitions including one or more derived geometries of sufficient quality for alignment with a digital definition of the composite component,

the metrology process generating a spatial framework for the composite component and the at least one physical object,

wherein the spatial framework comprises a point cloud defining a set of vertices representative of a first external surface of the composite component and a second external surface of the at least one physical object,

wherein the metrology process produces the digital definition of the composite component;

generating an affine transformation using relative alignments of the spatial framework of the composite component and the at least one physical object;

aligning, using the affine transformation, the point cloud with the digital definition of the composite component and the at least one physical object,

wherein aligning includes rotation and translation to make the point cloud substantially coincident with the digital definition,

wherein substantially coincident is defined as a result where a given subset of points from the point cloud are within a predetermined distance from the digital definition;

retrieving information related to the composite component including a number of layers of composite material;

selecting at least one composite layer for display;

generating in the processor unit a multi-dimensional representation of a combined augmented reality of the composite component including information of the at least one composite layer;

transmitting the multi-dimensional representation to the number of portable devices; and

displaying in the number of portable devices the multi-dimensional representation comprising physical data for an image of a surface of the composite component and digital data for a first overlay of the at least one composite layer beyond the surface of the composite component,

wherein the first overlay comprises inspection data related to inspection of the composite component after installation of the patch.

17. The method of claim 16 further comprising:

generating an affine transformation using a spatial relationship between the physical data and the digital data received from the physical data captured.

18. The method of claim 17 further comprising:

generating the multi-dimensional representation of a combined augmented reality using the affine transformation.

19. The method of claim 17, wherein the affine transformation includes a rigid motion.