Virtual reality and augmented reality for industrial automation

• US 10,388,075 B2
• Filed: 09/28/2017
• Issued: 08/20/2019
• Est. Priority Date: 11/08/2016
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Schmirler, Paul D., Nguyen, Thong T., Nicoll, Alex L., Vasko, David
• Primary Examiner(s): Chu, David H
• Application Number:
  US15/718,907
• Publication Number:
  US 20180130260A1
• Time in Patent Office: 691 Days
• Field of Search: None
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05B 23/0216 Human interface functionality, e.g. monitoring system providing help to the user in the selection of tests or in its configuration
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 10/0633 Workflow analysis
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32

Abstract

An industrial visualization system generates and delivers virtual reality (VR) and augmented reality (AR) presentations of industrial facilities to wearable appliances to facilitate remote or enhanced interaction with automation systems within the facility. VR presentations can comprise three-dimensional (3D) holographic views of a plant facility or a location within a plant facility. The system can selectively render a scaled down view that renders the facility as a 3D scale model, or as a first-person view that renders the facility as a full-scale rendition that simulates the user'"'"'s presence on the plant floor. Camera icons rendered in the VR presentation can be selected to switch to a live video stream generated by 360-degree cameras within the plant. The system can also render workflow presentations that guide users through the process of correcting detected maintenance issues.

Claims

1. A system, comprising:

a memory that stores executable components;

a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising;

a device interface component configured to receive industrial data from industrial devices associated with an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;

a client interface component configured to receive identity data, location data, and orientation data from a wearable appliance, wherein the identity data defines a user identity associated with the wearable appliance, and the location data and the orientation data indicate a current location and orientation, respectively, of the wearable appliance;

a monitoring component configured to identify occurrence of a maintenance issue based on a determination that a subset of the industrial data satisfies a condition indicative of the maintenance issue; and

a rendering component configured to, in response to identification of the maintenance issue, retrieve workflow data defining a workflow for correcting the maintenance issue, and generate augmented reality presentation data that renders, on the wearable appliance, an augmented reality presentation based on the workflow data, the identity data, the location data, and the orientation data,

wherein the client interface component is further configured to receive, from the wearable appliance, virtual note data defining an alphanumeric message, an indication of an industrial asset to which the alphanumeric message pertains, and a viewing permission criterion for the alphanumeric message, and

the rendering component is further configured to, in response to determining that a current location and a current orientation of a second wearable appliance causes the industrial asset to be within a field of view of the second wearable appliance, and that the second wearable appliance is associated with a user identity that satisfies the viewing permission criterion, render the alphanumeric message on the second wearable appliance.

2. The system of claim 1, wherein the rendering component is configured to select the wearable appliance for delivery of the augmented reality presentation data based on a correlation between the identity data and at least one of a type of the maintenance issue or a location of the maintenance issue.

3. The system of claim 1, wherein

the workflow data defines a sequence of steps to be performed to correct the maintenance issue, and

the rendering component is configured to, for a current step of the sequence of steps, render graphical indications on the wearable appliance that provide instruction regarding performance of the current step.

4. The system of claim 3, wherein the rendering component is configured to render the graphical indications at a location within a display area of the wearable appliance selected based on the location data and the orientation data.

5. The system of claim 3, wherein the graphical indications comprise at least one of an alphanumeric instruction, a graphical arrow rendered on a display screen of the wearable appliance indicating a direction toward the maintenance issue, a graphical indicator rendered at a position of the display screen that identifies an industrial device or a machine component on which the current step is to be performed, an indication that the current step has been correctly performed, or an indication that the current step has been incorrectly performed.

6. The system of claim 3, wherein the rendering component is further configured to render a video on the wearable appliance demonstrating performance of the current step.

7. The system of claim 1, wherein the rendering component is configured to select the workflow data from a library of defined workflows stored on the memory based on the maintenance issue.

8. The system of claim 1, wherein the maintenance issue is at least one of a deviation of a key performance indicator from a defined range, detection of a machine failure or a device failure, or an alarm condition of a machine or one of the industrial devices.

9. The system of claim 3, wherein

the monitoring component is further configured to detect, based on monitoring of at least one of the industrial data, the location data, or the orientation data, a deviation from the workflow, and

the rendering component is further configured to, in response to detection by the monitoring component of the deviation from the workflow, render a corrective instruction on the wearable device.

10. The system of claim 9, wherein the monitoring component is further configured to calculate a performance metric based on at least one of a number of deviations from the workflow or a speed of completion of the workflow, and to store the performance metric in association with the user identity.

11. The system of claim 1, wherein

the client interface component is further configured to receive, from the wearable appliance, an asset identifier of an industrial asset to be located, and

the rendering component is further configured to, in response to receipt of the asset identifier, render, on the wearable appliance based on the location data and the orientation data, one or more graphical indicators that indicate a direction or location of the industrial asset.

12. The system of claim 1, wherein

the client interface component is further configured to receive, from the wearable appliance, a spoken request for a current status of a specified industrial asset, and

the rendering component is configured to, in response to receipt of the spoken request, retrieve a subset of the industrial data pertaining to the current status of the specified asset and render, on the wearable appliance, the subset of the industrial data.

13. A method, comprising:

receiving, by a system comprising a processor, industrial data generated by industrial devices of an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;

receiving, by the system, identity data specifying a user identity associated with a wearable appliance, location data specifying a current location of the wearable appliance, and orientation data specifying a current orientation of the wearable appliance;

identifying, by the system, occurrence of a maintenance issue based on a determination that a subset of the industrial data satisfies a condition indicative of the maintenance issue;

in response to the identifying of the maintenance issue, retrieving, by the system, workflow data that defines a workflow for correcting the maintenance issue;

generating, by the system, augmented reality presentation data that renders, on the wearable appliance, an augmented reality presentation based on the workflow data, the identity data, the location data, and the orientation data;

receiving, by the system from the wearable appliance, virtual note data comprising an alphanumeric message, an indication of an industrial asset to which the alphanumeric message pertains, and a viewing permission criterion for the alphanumeric message; and

in response to determining that a current location and a current orientation of another wearable appliance places the industrial asset within a field of view of the other wearable appliance, and that the other wearable appliance is associated with a user identity that satisfies the viewing permission criterion, rendering, by the system, the alphanumeric message on the other wearable appliance.

14. The method of claim 13, further comprising selecting, by the system, the wearable appliance for delivery of the augmented reality presentation data based on a correlation between the identity data and at least one of a type of the maintenance issue or a location of the maintenance issue.

15. The method of claim 13, wherein the workflow data defines a sequence of steps to be performed to correct the maintenance issue, and the method further comprises:

for a current step of the sequence of steps, rendering, by the system, graphical indications on the wearable appliance that guide performance of the current step; and

in response to determining, based on a monitoring of at least one of the location data, the orientation data, the industrial data, or enterprise data that the current step is not performed correctly or is being performed in a manner that may prevent performance of a next step of the sequence of steps, rendering feedback data on the wearable appliance indicating that the current step is not being performed correctly.

16. The method of claim 15, wherein the rendering the graphical indications comprises rendering the graphical indications at a location on a display screen of the wearable appliance selected based on the location data and the orientation data.

17. The method of claim 15, wherein the rendering the graphical indications comprises rendering at least one of an alphanumeric instruction, a graphical arrow rendered on a display screen of the wearable appliance indicating a direction toward the maintenance issue, a graphical indicator rendered at a position of the display screen that identifies an industrial device or a machine component on which the current step is to be performed, an indication that the current step has been correctly performed, or an indication that the current step has been incorrectly performed.

18. The method of claim 15, wherein the rendering the graphical indications comprise rendering, on the wearable appliance, a video demonstrating performance of the current step.

19. A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:

receiving industrial data generated by industrial devices of an industrial facility;

receiving identity data identifying a user identity associated with a first wearable appliance;

receiving location data identifying a current location of the first wearable appliance;

receiving orientation data identifying a current orientation of the first wearable appliance;

identifying occurrence of a maintenance issue based on a determination that a subset of the industrial data satisfies a condition indicative of the maintenance issue;

in response to the identifying, retrieving workflow data that defines a workflow for correcting the maintenance issue;

rendering, on the first wearable appliance, an augmented reality presentation based on the workflow data, the identity data, the location data, and the orientation data;

receiving, from the first wearable appliance, virtual note data comprising an alphanumeric message, an indication of an industrial asset to which the alphanumeric message pertains, and a viewing permission criterion for the alphanumeric message; and

in response to determining that a current location and a current orientation of a second wearable appliance places the industrial asset within a field of view of the second wearable appliance, and that the second wearable appliance is associated with a user identity that satisfies the viewing permission criterion, rendering the alphanumeric message on the second wearable appliance.

20. The non-transitory computer-readable medium of claim 19, wherein the workflow data is selected based on at least one of an analysis of a subset of the industrial data relating to a machine for which the maintenance issue is identified, machine history data that records a maintenance history or an operating history of a machine for which the maintenance issue is identified, a current state of the machine, an analysis of enterprise data, or user information.