Augmented reality safety automation zone system and method

• US 10,445,944 B2
• Filed: 10/19/2018
• Issued: 10/15/2019
• Est. Priority Date: 11/13/2017
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Galera, Richard, Pritchard, John
• Primary Examiner(s): Chen, Frank S
• Application Number:
  US16/165,360
• Publication Number:
  US 20190147655A1
• Time in Patent Office: 361 Days
• Field of Search: None
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: G05B 19/4061 Avoiding collision or forbidden zones
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05B 2219/50193 Safety in general
• CPC Class Codes: G05B 9/02 electric
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/62 Semi-transparency
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

An industrial visualization system generates and delivers virtual reality (VR) and augmented reality (AR) presentations of industrial facilities to wearable appliances to facilitate remote or enhanced interaction with automation systems within the facility. The presentation system can also leverage safety zone definition data to visualize graphical representations of defined safety zones within a line of sight of the wearable appliance. The safety zones can be rendered as three-dimensional semi-transparent overlays corresponding to the defined dimensions and location of the safety zone relative to its corresponding safety sensor. By visualizing defined safety zones as overlays within a wearer'"'"'s field of view as part of an AR presentation, the presentation system can serve as a visual guide that helps the wearer to avoid inadvertently entering the safety zones, thereby reducing machine downtime due to accidental initiation of safety actions.

Claims

1. A system, comprising:

a memory that stores executable components;

a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising;

a device interface component configured to receive industrial data from industrial devices associated with an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;

a client interface component configured to receive user location data and user orientation data from a client device, wherein the user location data and the user orientation data indicate a current location and orientation, respectively, of the client device; and

a rendering component configured to generate augmented reality presentation data that renders an augmented reality presentation of the industrial facility on a wearable appliance,

wherein

the device interface component is further configured to receive safety zone configuration data from a safety sensor device, the safety zone configuration data defining dimensions of a safety field defined for the safety sensor device, and

the rendering component is further configured to, in response to determining that the user location data and the user orientation data indicate that a field of view of the client device encompasses a portion of the safety field, render, on the augmented reality presentation based on the safety zone configuration data, a graphical representation of the portion of the safety field.

2. The system of claim 1, wherein the device interface component is further configured to receive safety sensor location data indicating a location of the safety sensor device and safety sensor orientation data indicating an orientation of the safety sensor device, and the rendering component is further configured to render the graphical representation of the portion of the safety field based on the safety zone configuration data, the safety sensor location data, and the safety sensor orientation data correlated with the user location data and the user orientation data.

3. The system of claim 2, wherein the graphical representation of the portion of the safety field comprises a semi-transparent three-dimensional shape rendered at a location within the field of view corresponding to the location of the safety sensor device indicated by the safety sensor location data.

4. The system of claim 2, wherein the rendering component is further configured to, in response to a change to at least one of the safety sensor location data or the safety sensor orientation data, update at least one of a location of the graphical representation within the field of view or an orientation of the graphical representation within the field of view in accordance with the change.

5. The system of claim 2, wherein the client interface component is further configured to determine whether the user location data corresponds to a location within the safety field based on a correlation of the user location data, the safety zone configuration data, the safety sensor location data, and the safety sensor orientation data.

6. The system of claim 5, wherein the device interface component is further configured to, in response to a determination by the client interface component that the user location data corresponds to the location within the safety field and that a defined safety function has not been performed by the safety sensor device, generate a control output that places an industrial system in a safe state.

7. The system of claim 5, wherein

the safety zone configuration data further defines dimensions of one or more warning fields defined for the safety sensor device, and

the client interface component is further configured to, in response to determining that the user location data corresponds to a location within a warning field of the one or more warning fields, send at least one of a visual indication, an audible indication, or a tactile indication to the client device.

8. The system of claim 1, wherein the safety zone configuration data defines dimensions of multiple safety fields defined for the safety sensor device, and the graphical representation renders the multiple safety fields using respective different colors.

9. The system of claim 5, wherein the device interface component is further configured to, in response to a determination by the client interface component that the user location data corresponds to the location within the safety field, increase a brightness of the graphical representation of the portion of the safety field.

10. The system of claim 1, wherein the client device is at least one of a wearable appliance or a hand-held client device.

11. A method, comprising:

receiving, by a system comprising a processor, industrial data generated by industrial devices of an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;

receiving, by the system, user location data and user orientation data from a wearable appliance, wherein the user location data and the user orientation data indicate a current location and orientation, respectively, of the wearable appliance;

generating, by the system, augmented reality presentation data that renders an augmented reality presentation of the industrial facility on a wearable appliance;

receiving, by the system, safety zone configuration data from a safety sensor device, the safety zone configuration data defining dimensions of a safety field defined for the safety sensor device; and

in response to determining that the user location data and the user orientation data indicate that a portion of the safety field is within a field of view of the wearable appliance, rendering, by the system on the augmented reality presentation based on the safety zone configuration data, a graphical representation of the portion of the safety field.

12. The method of claim 11, further comprising:

receiving, by the system, safety sensor location data indicating a location of the safety sensor device and safety sensor orientation data indicating an orientation of the safety sensor device; and

rendering, by the system, the graphical representation of the portion of the safety field based on the safety zone configuration data, the safety sensor location data, and the safety sensor orientation data correlated with the user location data and the user orientation data.

13. The method of claim 12, wherein the rendering the graphical representation of the portion of the safety field comprises rendering the graphical representation of the portion of the safety field as a semi-transparent three-dimensional shape rendered at a location within the field of view corresponding to the location of the safety sensor device indicated by the safety sensor location data.

14. The method of claim 12, further comprising:

in response to detecting a change to at least one of the safety sensor location data or the safety sensor orientation data, updating, by the system, at least one of a location of the graphical representation within the field of view or an orientation of the graphical representation within the field of view in accordance with the change.

15. The method of claim 12, further comprising determining, by the system based on a correlation of the user location data, the safety zone configuration data, the safety sensor location data, and the safety sensor orientation data, that the user location data corresponds to a location within the safety field.

16. The method of claim 15, further comprising:

in response to determining that the user location data corresponds to the location within the safety field and that a defined safety function has not been performed by the safety sensor device, generating, by the system, a control output configured to place an industrial machine in a safe state.

17. The method of claim 15, wherein

the safety zone configuration data further defines dimensions of a warning field defined for the safety sensor device, and

the method further comprises,

in response to determining that the user location data corresponds to a location within the warning field, sending, by the system, at least one of a visual indication, an audible indication, or a tactile indication to the wearable appliance.

18. The method of claim 11, wherein

the safety zone configuration data defines dimensions of multiple fields of a monitored safety zone defined for the safety sensor device,

the multiple fields comprise at least one warning field and at least one safety field including the safety field, and

the method further comprises:

in response to determining that the user location data and the user orientation data indicate that portions of at least two of the multiple fields are within the field of view of the wearable appliance, rendering, by the system on the augmented reality presentation based on the safety zone configuration data, the portions of the multiple fields in respective different colors.

19. A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:

receiving industrial data generated by industrial devices of an industrial facility, the industrial devices comprising at least an industrial controller that receives input data from at least one industrial input device and sends output data to one or more industrial output devices;

receiving user location data and user orientation data from a wearable appliance, wherein the user location data and the user orientation data indicate a current location and orientation, respectively, of the wearable appliance;

generating augmented reality presentation data that renders an augmented reality presentation of the industrial facility on a wearable appliance;

receiving safety zone configuration data from a safety sensor device, the safety zone configuration data defining dimensions of a safety field defined for the safety sensor device; and

in response to determining that the user location data and the user orientation data indicate that a portion of the safety field is within a field of view of the wearable appliance, rendering, on the augmented reality presentation based on the safety zone configuration data, a graphical representation of the portion of the safety field.

20. The non-transitory computer-readable medium of claim 19, wherein the operations further comprise:

receiving safety sensor location data indicating a location of the safety sensor device and safety sensor orientation data indicating an orientation of the safety sensor device; and

rendering the graphical representation of the portion of the safety field based on the safety zone configuration data, the safety sensor location data, and the safety sensor orientation data correlated with the user location data and the user orientation data.