Augmented reality presentation of an industrial environment

• US 10,319,128 B2
• Filed: 12/27/2016
• Issued: 06/11/2019
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Rockwell Automation, Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Rockwell Automation, Inc.)
• Inventors: Billi-Duran, Sharon M., Como, Christopher W., Gray, Edward A., Reissner, Kyle K., Walter, Jonathan D., Singhai, Mohit, Reichard, Douglas J., Sandler, Scott N., Bliss, Ronald E., Pantaleano, Michael J., Cahalane, Ryan, Korpela, Jessica L., McCleave, Jr., Bruce T.
• Primary Examiner(s): Chen, Frank S
• Application Number:
  US15/391,213
• Publication Number:
  US 20180089870A1
• Time in Patent Office: 896 Days
• Field of Search: None
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05B 23/0205 by means of a monitoring system capable of detecting and responding to faults
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip

Abstract

A industrial visualization system generates augmented reality presentations for rendering on a user'"'"'s wearable computer as the user traverses an industrial facility. The system can leverage data indexed in a federated data model that unifies plant-wide data from multiple diverse sources under a common namespace. By leveraging the unified plant data and interacting with a wearable computer worn by a user, the augmented reality presentation system provides automation system data, notifications, and proactive guidance to a user via modification of the user'"'"'s view of his or her immediate surroundings. Such modifications can include superimposing data values or indicators on the user'"'"'s view of a machine or automation system through the user'"'"'s wearable computer. The system can customize presentation of this information based on the user'"'"'s role, location, line of sight, type of wearable device, and/or other contextual information.

Claims

1. A system for generation and delivery of augmented reality presentations, comprising:

a memory that stores executable components;

a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising;

a device interface component configured to receive contextual data from a client device, the contextual data comprising at least location data identifying a location within an industrial environment and orientation data identifying an orientation of the client device;

a search component configured to, in response to a determination that the contextual data indicates that an industrial automation system is within a field of view of the client device, identify data items that are relevant to the industrial automation system, wherein a first copy of the data items are stored on an industrial device of the industrial automation system and a second copy of the data items are stored in a federated data model on the memory; and

a visualization component configured to generate an augmented reality presentation that renders the data items on the client device as respective overlays on the field of view,

wherein

the visualization component is further configured to;

while the client device is within a range of near field communication with the industrial device, select, as a source of the data items to be rendered on the augmented reality presentation, the first copy of the data items stored on the industrial device, retrieve the first copy of the data items, and render the first copy of the data items on the client device, and

while the client device is not within the range of near field communication with the industrial device, select, as the source of the data items to be rendered on the augmented reality presentation, the second copy of the data items stored in the federated data model, retrieve the second copy of the data items from the federated data model, and render the second copy of the data items on the client device, and

the device interface component is further configured to send the augmented reality presentation to the client device.

2. The system of claim 1, wherein the client device is a wearable computer.

3. The system of claim 1, the executable components further comprising:

a monitoring component configured to determine that one or more data items stored in the federated data model satisfy a defined criterion indicative of a performance issue relating to an industrial asset; and

a notification component configured to select a recipient for notification of the performance issue based on one or more notification rules, and to initiate delivery of a notification to a client device associated with the recipient via another augmented reality presentation by the visualization component.

4. The system of claim 3, wherein the other augmented reality presentation comprises at least one of a directional icon that indicates a direction to a source of the performance issue relative to a current location of the client device associated with the recipient or a color-coded overlay positioned near a source of the performance issue.

5. The system of claim 3, wherein

the device interface is further configured to receive environment data from the client device, and

the monitoring component is further configured to, in response to a determination that the environment data is indicative of a hazard or a performance problem of the industrial automation system, initiate delivery of another augmented reality presentation to the client device, the other augmented reality presentation rendering information about the hazard or performance issue.

6. The system of claim 5, wherein the environment data comprises at least one of infrared data;

heat signature data;

vibration data;

ambient noise level data;

flux data;

or data indicative of presence of a gas, particulates, smoke, or a toxin.

7. The system of claim 1, wherein

the visualization component is configured to position a data item, of the data items, at a location within the augmented reality presentation that causes the data item to be positioned on or near a corresponding industrial device within the field of view of the client device, and

the location within the augmented reality presentation is determined based at least in part on the location data and the orientation data.

8. The system of claim 1, wherein the visualization component is further configured to select the data items for presentation on the augmented reality presentation based in part on a user role associated with the client device.

9. The system of claim 1, wherein the visualization component is configured to, in response to a determination that a data item, of the data items, has a value that is outside a defined range of values indicative of normal operation of the industrial automation system, render the data item on the augmented reality presentation in a manner that distinguishes the data item from other data items rendered on the augmented reality presentation.

10. A method for generating augmented reality presentations for rendering of industrial data, comprising:

receiving, by a system comprising at least one processor, contextual data from a client device, wherein the contextual data comprises at least location data identifying a location of the client device within an industrial environment and orientation data identifying an orientation of the client device;

in response to determining that the contextual data signifies that an automation system is within a field of view of the client device;

identifying, by the system, data items that are relevant to the automation system,

selecting, by the system, a source of the data items from among a first copy of the data items stored on an industrial device of the automation system or a second copy of the data items stored on a federated data model maintained on a cloud platform, wherein the selecting comprises;

while the client device is within a range of a near field communication link with the industrial device, selecting, as the source of the data items, the first copy of the data items stored on the industrial device, and

while the client device is not within the range of the near field communication link with the industrial device, selecting, as the source of the data items, the second copy of the data items from the federated data model,

retrieving, by the system, the data items from the source of the data items,

generating, by the system, an augmented reality presentation that renders the data items on the client device as respective overlays on the field of view; and

sending, by the system, the augmented reality presentation to the client device.

11. The method of claim 10, wherein the receiving the contextual data comprises receiving the contextual data from a wearable computer.

12. The method of claim 10, further comprising:

determining, by the system, that one or more data items maintained in the federated data model satisfy a defined criterion indicative of a performance issue relating to an industrial asset; and

in response to the determining;

selecting, by the system, a recipient for notification of the performance issue based on one or more notification rules, and

sending, by the system, a notification to a client device associated with the recipient via another augmented reality presentation.

13. The method of claim 12, wherein the sending the notification comprises rendering one or more directional icons on the other augmented reality presentation, the directional icons indicating a direction to a source of the performance issue relative to a current location of the recipient.

14. The method of claim 10, wherein

the generating the augmented reality presentation comprises orienting a data item, of the data items, at a location on the augmented reality presentation that causes the data item to be positioned on or near a corresponding industrial device within the field of view of the client device, and

the location on the augmented reality presentation is determined based at least in part on the location data and the orientation data.

15. The method of claim 10, wherein the searching comprises selecting at least one of the data items based at least in part on a user role associated with the client device.

16. The method of claim 10, further comprising:

determining, by the system, that a data item, of the data items, associated with an industrial device has a value that is outside a defined tolerance indicative of normal operation of the industrial device; and

in response to the determining, rendering, by the system, the data item on the augmented reality presentation using at least one of a different color, a different size, or a different font relative to other data items rendered on the augmented reality presentation.

17. The method of claim 10, further comprising:

monitoring, by the system, a speed of motion of the client device; and

in response to determining, based on the monitoring, that the speed exceeds a defined speed threshold, removing the data items from the field of view.

18. A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:

receiving contextual data from a client device, the contextual data comprising at least location data that identifies a location of the client device within an industrial environment and orientation data identifying an orientation of the client device;

in response to determining that the contextual data indicates that an automation system is within a field of view of the client device, retrieving data items that are determined to be relevant to the automation system, wherein the retrieving comprises;

while the client device is within a range of near field communication with an industrial device that stores a first copy of the set of data items, retrieving, as the data items, the first copy of the data items from the industrial device, and

while the client device is outside the range of the near field communication with the industrial device, retrieving, as the data items, a second copy of the set of data items stored in a federated data model maintained on a cloud platform;

generating an augmented reality presentation that renders the data items on the client device as respective overlays on the field of view; and

sending the augmented reality presentation to the client device.

19. The non-transitory computer-readable medium of claim 18, the operations further comprising:

in response to determining that a data item maintained in the federated data model has a value indicative of a performance issue relating to an industrial asset;

selecting a recipient for notification of the performance issue based on one or more notification rules; and

sending a notification to a client device associated with the recipient via another augmented reality presentation.

20. The non-transitory computer-readable medium of claim 19, wherein the sending the notification comprises rendering one or more directional icons on the other augmented reality presentation, the one or more directional icons indicating a direction to a source of the performance issue relative to a current location of the client device associated with the recipient.