Portable augmented reality

• US 8,902,254 B1
• Filed: 09/02/2010
• Issued: 12/02/2014
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Laughlin, Brian Dale, Costello, John, Vandenbrande, Jan H., Murrish, Richard Eugene
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): McCulley, Ryan D
• Application Number:
  US12/874,586
• Time in Patent Office: 1,552 Days
• US Class Current: 345/633
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

The different advantageous embodiments provide a system comprising a processor unit, a sensor system, and a user interface. The processor unit is configured to execute a data manipulation process. The sensor system is configured to retrieve physical data about a number of physical objects. The user interface has data manipulation controls for the data manipulation process.

Claims

1. An apparatus for providing an augmented reality view of an object comprising:

a processor unit configured to execute a data manipulation process;

a sensor system configured to retrieve physical data about a physical object, the sensor system comprising a number of imaging devices configured to obtain physical data for the physical object, the physical data including an image of a surface of the physical object, the sensor system further including a positioning system, a clock, and an accelerometer;

a handheld device, the sensor system associated with the handheld device, and the processor unit positioned remotely from the handheld device, the handheld device configured to transmit the physical data to the processor unit; and

a user interface associated with the handheld device having data manipulation controls for the data manipulation process;

wherein the processor unit is configured to generate a number of digital definitions in a metrology process using the physical data, the number of digital definitions including one or more derived geometries of sufficient quality to align with a digital definition of the object, generate an affine transformation in a data compilation process using a spatial relationship between the physical data and digital data associated with the object in the processor unit, wherein the affine transformation includes a translation and a rotation of a point cloud to be substantially coincident with a digital definition, and generate in a multi-dimensional generation process in the processor unit a multi-dimensional representation of a combined augmented reality of the object using the affine transformation, the multi-dimensional representation comprising physical data for the physical object including the image of the surface of the object and digital data for the physical object including a first overlay representing components associated with the object and positioned below the surface and a second overlay indicating a repaired area of the object, the processor unit further configured to transmit the multi-dimensional representation to the handheld device;

wherein the object comprises a composite structure having a number of plies positioned under a surface of the composite structure and further comprising selecting a ply, via the user interface, in the number of plies to display in the multi-dimensional representation, and displaying the ply and a repair to the ply as a result of the selection.

2. The apparatus of claim 1, wherein the sensor system further includes a compass.

3. The apparatus of claim 1, wherein the data manipulation controls provide rotation and scaling of a multi-dimensional representation displayed on the user interface.

4. The apparatus of claim 1 further comprising a communications unit, the communications unit configured to receive the multidimensional representation.

5. An apparatus for providing an augmented reality view of an aircraft comprising:

a number of portable devices, wherein the number of portable devices is configured to obtain physical data for an object using a sensor system associated with the number of portable devices, the object comprising an aircraft having a composite structure, transmit the physical data, and receive a multi-dimensional representation, the sensor system further including a number of imaging systems to obtain the physical data, a positioning system, a clock, and an accelerometer; and

a computer system remote from the number of portable devices configured to receive the physical data generated by the number of portable devices, generate a number of digital definitions in a metrology process using the physical data, the number of digital definitions including one or more derived geometries of sufficient quality to align with a digital definition of the object, generate an affine transformation in a data compilation process using a spatial relationship between the physical data and digital data associated with the object in the processor unit, wherein the affine transformation includes a translation and a rotation of a point cloud to be substantially coincident with a digital definition, and generate in a multi-dimensional generation process in the processor unit a multi-dimensional representation of a combined augmented reality of the object using the affine transformation, and transmit the multi-dimensional representation to the number of portable devices for display, the multi-dimensional representation comprising the physical data and the digital data for the object, wherein the physical data includes an image of a surface of the composite structure and the digital data comprises a first overlay showing a repair to the composite structure below the surface of the composite structure and a second overlay showing a digital definition geometry of a structure below the surface of the composite structure;

wherein the number of portable devices include at least one user interface, and wherein the multi-dimensional representation is viewed by a user on the user interface;

wherein the composite structure has a number of plies positioned under a surface of the composite structure and further comprising selecting a ply, via the user interface, in the number of plies to display in the multi-dimensional representation, and displaying the ply and a repair to the ply as a result of the selection.

6. The apparatus of claim 5 further comprising:

a number of storage devices in the number of portable devices, the number of storage devices configured to store a number of multi-dimensional representations sent to the number of portable devices by the computer system.

7. The apparatus of claim 5 further comprising a metrology system in the computer system, the metrology system configured to generate a spatial framework for the object.

8. The apparatus of claim 7, wherein the spatial framework comprises a tessellated mesh.

9. The apparatus of claim 5 wherein the repair includes a time stamp.

10. A method for augmenting reality, the method comprising:

capturing physical data for an object at a point of use using a number of portable devices;

transmitting the physical data captured by the number of portable devices to a processor unit remote from the number of portable devices for alignment and integration;

generating a number of digital definitions in a metrology process in the processor unit using the physical data, the number of digital definitions including one or more derived geometries of sufficient quality to align with a digital definition of the object;

generating an affine transformation in a data compilation process in the processor unit using a spatial relationship between the physical data and digital data associated with the object in the processor unit, wherein the affine transformation includes a translation and a rotation of a point cloud to be substantially coincident with a digital definition;

generating in a multi-dimensional generation process in the processor unit a multi-dimensional representation of a combined augmented reality of the object using the affine transformation;

transmitting the multi-dimensional representation to the number of portable devices; and

displaying in the number of portable devices the multi-dimensional representation comprising physical data related to the object and digital data related to the object, the physical data comprising an image of a surface of the object and the digital data comprising a first overlay of damage areas beyond the surface of the object and a second overlay of a digital definition geometry of a structure beyond the surface of the object;

wherein the number of portable devices include at least one user interface, and wherein the multi-dimensional representation is viewed by a user on the user interface;

wherein the object comprises a composite structure having a number of plies positioned under a surface of the composite structure and further comprising selecting a ply, via the user interface, in the number of plies to display in the multi-dimensional representation, and displaying the ply and a repair to the ply as a result of the selection.

11. The method of claim 10, wherein the multi-dimensional representation is augmented reality.

12. The method of claim 10, wherein the affine transformation includes a rigid motion.

13. The method of claim 10 further comprising:

storing the multi-dimensional representation on the number of portable devices; and

retrieving the multi-dimensional representation stored on the number of portable devices.

14. The method of claim 10 further comprising displaying maintenance information related to the object in the multi-dimensional representation, the maintenance information comprising a plurality of time stamped repair data.

15. The method of claim 10, wherein the object comprises a composite material, and further comprising displaying overlays of repairs associated with the damage areas for the object in the multi-dimensional representation.