System and method for augmented reality display of dynamic environment information

• US 9,947,138 B2
• Filed: 04/14/2015
• Issued: 04/17/2018
• Est. Priority Date: 04/15/2014
• Status: Active Grant

• Current Assignee: Huntington Ingalls, Inc. (Huntington Ingalls Industries Incorporated)
• Original Assignee: Huntington Ingalls, Inc. (Huntington Ingalls Industries Incorporated)
• Inventors: Bare, Brian, Martin, Jonathan, Ryan, Patrick, Sells, Paul, Lawrence, Mark, Brooks, Justin, Mason, Joshua
• Primary Examiner(s): Good Johnson, M
• Application Number:
  US14/686,427
• Publication Number:
  US 20150294506A1
• Time in Patent Office: 1,099 Days
• Field of Search: None
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G08B 21/12 responsive to undesired emission of substances, e.g. pollution alarms
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A method for providing environment information to a mobile device user is presented. The method comprises receiving a request for target environment information from a mobile device, determining the pose of the mobile interface device relative to the target environment, and obtaining target environment data for one or more measurable environment parameters (e.g., radiation level). The target environment data is used to assemble augmented reality information configured for viewing in conjunction with a real-time view of the target environment captured by the mobile interface device. The target augmented reality information is then transmitted to the mobile device for display to the user.

Claims

1. A method for providing environment information to a mobile interface device user in a dynamic structural environment in the form of a dynamic structure having an associated environment coordinate system, the method comprising:

receiving on a central data processor from the mobile interface device over a communication network a request for target environment information, the request including request information usable by the central data processor to determine the pose of the mobile interface device relative to a target environment that is a portion of the dynamic structural environment;

receiving mobile interface device location information from a local positioning system of the dynamic structure, said location information being provided using the environment coordinate system;

determining the pose of the mobile interface device relative to the target environment using the request information and the mobile interface device location information;

obtaining target environment data for one or more measurable environment parameters;

assembling augmented reality (AR) environment information for transmission to and display on the mobile interface device, the AR environment information being assembled using the target environment data and being configured for viewing in conjunction with a real-time view of the target environment captured by the mobile interface device; and

transmitting the AR environment information to the mobile interface device,

wherein the action of determining the pose of the mobile interface device accounts for spatial changes in the environment coordinate system in the target environment due to changes in the dynamic structural environment.

2. A method according to claim 1, wherein the mobile interface device is one of the set consisting of a tablet computer, a smartphone, and a wearable heads-up display.

3. A method according to claim 1 wherein the request for target environment information includes a digital image of the target area captured by the mobile interface device.

4. A method according to claim 1 wherein the action of obtaining target environment data includes:

receiving environment parameter data from one or more sensors located in or adjacent the target environment.

5. A method according to claim 4 wherein the AR environment information includes a graphical representation of at least one of the one or more measurable environment parameters, the graphical representation having been constructed using the target environment data.

6. A method according to claim 1 wherein the target environment data comprises radiation sensor data from at least one location in or adjacent the target environment.

7. A method according to claim 6 further comprising:

constructing a three dimensional estimate of radiation levels in the target environment.

8. A method according to claim 7 wherein the AR environment information includes a graphical representation of the three dimensional estimate of radiation levels in the target environment that can be superimposed over the real-time view of the target environment captured by the mobile interface device.

9. A method according to claim 1 wherein the actions of determining, obtaining, assembling and transmitting are periodically repeated.

10. An automated system for providing environment information to a mobile interface device in a dynamic structural environment in the form of a dynamic structure having an environment coordinate system, the system comprising:

at least one mobile interface device configured for variable disposition within the dynamic structural environment, capturing information about a target environment within the structural environment, and transmitting the target area information over a network;

a domain coordinate management system comprising one or more data processors configured for receiving information on a current conformation of the dynamic structural environment and resolving spatial differences in the environment coordinate system between the current conformation of the dynamic structural environment and a stored model of the dynamic structural environment;

a local positioning system disposed at least partially within the dynamic structure, the local positioning system being in communication with the at least one mobile interface device and the domain coordinate system via the network and being configured to provide location information usable to determine a mobile interface device pose, said mobile interface device pose comprising a position and orientation of the mobile interface device relative to the target environment in the environment coordinate system; and

a central processor comprising at least one data processing machine in communication with the at least one mobile interface device, the domain coordinate management system, and the local positioning system via the network, the central processor being configured for

receiving from a requesting one of the at least one mobile interface device a request for target environment information, the request including information usable to determine the pose of the requesting mobile interface device relative to the target environment,

determining the pose of the requesting mobile interface device relative to the target environment, accounting for spatial changes in the environment coordinate system in the target area due to changes in the dynamic structural environment,

obtaining target environment data for one or more measurable environment parameters,

assembling augmented reality (AR) environment information for transmission to and display on the requesting mobile interface device, the AR environment information being assembled using the target environment data and being configured for viewing in conjunction with a real-time view of the target environment captured by the requesting mobile interface device, and

transmitting the AR environment information to the requesting mobile interface device.

11. A system according to claim 10 further comprising:

at least one environment data system comprising one or more sensors located in or adjacent the target environment, the one or more sensors being configured for capturing the target environment data.

12. A system according to claim 11 wherein the one or more sensors are radiation sensors and the target environment data comprises a radiation level from at least one location in or adjacent the target environment.

13. A system according to claim 10 wherein the domain coordinate management system comprises one or more computers in communication with the central processor and the local positioning system and is being configured for

establishing and maintaining spatial addresses for spaces and points within the dynamic structural environment, and

receiving spatial address information from both the local positioning system and from the stored model of the dynamic structural environment.

14. A method for providing radiation environment information to a mobile interface device user in a dynamic structural environment in the form of a dynamic structure having an associated environment coordinate system, the method comprising:

receiving on a central data processor from the mobile interface device over a communication network a request for target area radiation information, the request including request information usable by the central data processor to determine a pose of the mobile interface device relative to a target area in the dynamic structural environment;

receiving mobile interface device location information from a local positioning system of the dynamic structure, said location information being provided using the environment coordinate system;

determining the pose of the mobile interface device relative to the target area using the request information and the mobile interface device location information;

obtaining target area radiation information;

assembling augmented reality (AR) radiation environment information for transmission to and display on the mobile interface device, the AR radiation environment information being assembled using the target area radiation information and being configured for viewing in conjunction with a real-time view of the target area captured by the mobile interface device; and

transmitting the AR radiation environment information to the mobile interface device,

wherein the action of determining the pose of the mobile interface device accounts for spatial changes in the environment coordinate system in the target area due to changes in the dynamic structural environment.

15. A method according to claim 14, wherein the mobile interface device is one of the set consisting of a tablet computer, a smartphone, and a wearable heads-up display.

16. A method according to claim 14 wherein the request for target environment information includes a digital image of the target area captured by the mobile interface device.

17. A method according to claim 14 wherein the action of obtaining radiation data includes:

receiving radiation data from one or more radiation sensors located in or adjacent the target environment.

18. A method according to claim 17 further comprising:

constructing a three dimensional estimate of radiation levels in the target area.

19. A method according to claim 18 wherein the AR radiation environment information includes a graphical representation of the three dimensional estimate of radiation levels in the target area that can be superimposed over the real-time view of the target area captured by the mobile interface device.