System and method for augmented reality discrepancy determination and reporting

• US 10,504,294 B2
• Filed: 11/26/2018
• Issued: 12/10/2019
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Huntington Ingalls, Inc. (Huntington Ingalls Industries Incorporated)
• Original Assignee: Huntington Ingalls, Inc. (Huntington Ingalls Industries Incorporated)
• Inventors: Bare, Brian, Martin, Jonathan, Ryan, Patrick, Lawrence, Mark, Sells, Paul
• Primary Examiner(s): Le, Michael
• Application Number:
  US16/199,780
• Publication Number:
  US 20190096138A1
• Time in Patent Office: 379 Days
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences

Abstract

A method is provided for providing target object information to a mobile interface device user in a dynamic structural environment. The method includes receiving from the mobile interface device a request for target object information associated with a target object in the dynamic structural environment. A pose of the mobile interface device relative to the target object is determined accounting for spatial differences in the environment coordinate system resulting from changes in the dynamic structure. The method also includes assembling AR target object information for transmission to and display on the mobile interface device and transmitting the AR target object information to the mobile interface device for display in conjunction with a real-time view of the target object.

Claims

1. A method for providing target object information to a mobile interface device user in a dynamic structural environment in the form of a dynamic structure having an environment coordinate system, the method comprising:

receiving on a central data processor from the mobile interface device over a communication network a request for target object information associated with a target object in the dynamic structural environment;

determining a pose of the mobile interface device relative to the target object accounting for spatial differences in the environment coordinate system resulting from changes in the dynamic structure;

assembling augmented reality (AR) target object information for transmission to and display on the mobile interface device; and

transmitting the AR target object information to the mobile interface device for display in conjunction with a real-time view of the target object.

2. The method of claim 1 further comprising:

receiving first spatial information from a local positioning system and second spatial information from a three dimensional model of the dynamic structure; and

using the first and second spatial information to determine the spatial differences in the environment coordinate system resulting from changes in the dynamic structure.

3. The method of claim 1 further comprising:

receiving a digital image of the target object captured by the mobile interface device; and

receiving mobile interface device location information from a local positioning system, said location information being provided in the environment coordinate system,

wherein the location information and the digital image are used in the action of determining the pose of the mobile interface device.

4. The method of claim 1 wherein the AR target object information includes at least one of the set consisting of information on a desired target object parameter, and

a graphical representation of a desired target object configuration.

5. The method of claim 1 further comprising:

receiving, from a measurement data system, target object data for one or more target object parameters.

6. The method of claim 5 wherein the AR target object information includes a graphical representation of the one or more target object parameters, the graphical representation being constructed using the target object data.

7. The method of claim 5 further comprising:

comparing the target object data to desired target object values for the one or more target object parameters to determine target object parameter deviations.

8. The method of claim 7 wherein the AR target object information includes a graphical representation of the target object parameter deviations.

9. The method of claim 7 further comprising:

performing at least one of the set consisting of a maintenance operation and a discrepancy correction operation on the target object; and

repeating the actions of receiving target object data, comparing the target object data to desired target object values to determine target object parameter deviations, assembling AR target object information, and transmitting the AR target information to the mobile interface device for display.

10. The method of claim 1, wherein the mobile interface device is one of the set consisting of a tablet computer, a smartphone, and a wearable heads-up display.

11. The method of claim 1 wherein the request for target environment information includes a digital image of the target object captured by the mobile interface device.

12. The method of claim 1 wherein the dynamic structure defines an interior space in which the target object is disposed.

13. The method of claim 1 wherein the dynamic structure comprises the target object.

14. The method of claim 1 wherein the spatial changes result from flexure of the dynamic structure.

15. An automated system for providing target object information to a mobile interface device for use in inspecting a target object in a dynamic structural environment in the form of a dynamic structure having an environment coordinate system, the system comprising:

at least one mobile interface device configured for variable disposition within the dynamic structural environment, capturing information about the target object within the structural environment, and transmitting the target object information over a network;

a local positioning system in communication with the at least one mobile interface device via the network; and

a central processor comprising at least one data processing machine in communication with the at least one mobile interface device and the local positioning system via the network, the central processor being configured for

receiving from a requesting one of the at least one mobile interface device a request for target object information associated with the target object;

determining a pose of the requesting mobile interface device relative to the target object accounting for spatial differences in the environment coordinate system resulting from changes in the dynamic structure;

assembling augmented reality (AR) target object information for transmission to and display on the mobile interface device; and

transmitting the AR target object information to the mobile interface device for display in conjunction with a real-time view of the target object.

16. The automated system of claim 15 wherein the central processor is also configured for

receiving first spatial information from a local positioning system and second spatial information from a three dimensional model of the dynamic structure;

and

using the first and second spatial information to determine the spatial differences in the environment coordinate system resulting from changes in the dynamic structure.

17. The automated system of claim 15 further comprising:

a measurement data system configured for capturing target object data for one or more target object parameters.

18. The automated system of claim 17,

wherein the central processor is also configured for

receiving the target object data, and

comparing the target object data to desired target object values for the one or more target object parameters to determine target object parameter deviations, and

wherein the AR target object information includes a graphical representation of the target object parameter deviations.