SYSTEMS AND METHODS FOR PRESENTING AN AUGMENTED REALITY

• US 20160284128A1
• Filed: 06/30/2015
• Published: 09/29/2016
• Est. Priority Date: 03/27/2015
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Michalscheck, Jimi R., Korpela, Jessica L., Reissner, Kyle K., Vasko, David A., Fordenwalt, Matthew W., Jauquet, John J., Ericsson, Matthew R., Wilber, Andrew, Michalscheck, Kelly A.
• Granted Patent: US 10,950,051 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G05B 19/409 characterised by using manual data input [MDI] or by using control panel, e.g. controlling functions with the panel; characterised by control panel details or by setting parameters G05B19/408, G05B19/4093 take precedence
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05B 2219/33286 Test, simulation analysator
• CPC Class Codes: G05B 2219/36167 Use camera of handheld device, pda, pendant, head mounted display
• CPC Class Codes: G05B 23/0216 Human interface functionality, e.g. monitoring system providing help to the user in the selection of tests or in its configuration
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A first computing device in an industrial automation system may include an image sensor that captures a visual representation of a portion of a facility including a first industrial automation equipment as image data and a communication component that communicatively couples the first computing device to a second computing device via a communication network, in which the first computing device receives relevant information related to the first industrial automation equipment from the second computing device. The first computing may include a display component that displays a plurality of real objects that visually represent physical features in the portion of the facility based at least in part on the image data, in which the plurality of real objects include a first real object that visual represents physical features of the first industrial automation equipment; and that displays a first virtual object superimposed on at least a portion of the plurality of real objects based at least in part on the relevant information.

Claims

1. A first computing device in an industrial automation system comprising:

an image sensor configured to capture a visual representation of a portion of a facility comprising a first industrial automation equipment as image data;

a communication component configured to communicatively couple the first computing device to a second computing device via a communication network, wherein the first computing device is configured to receive relevant information related to the first industrial automation equipment from the second computing device; and

a display component configured to;

display a plurality of real objects configured to visually represent physical features in the portion of the facility based at least in part on the image data, wherein the plurality of real objects comprises a first real object configured to visual represent physical features of the first industrial automation equipment; and

display a first virtual object superimposed on at least a portion of the plurality of real objects based at least in part on the relevant information.

2. The first computing device of claim 1, comprising a location sensor configured to determine a location and an orientation of the first computing device with respect to physical features of the facility;

wherein the display component is configured to;

display the first real object when the first computing device is oriented toward the first industrial automation equipment and is located within a certain distance from the first industrial automation equipment; and

display the first virtual object when the first real object is displayed.

3. The first computing device of claim 1, wherein the display component is configured to display the first virtual object in response to receiving a user selection of the first real object.

4. The first computing device of claim 1, wherein the display component is configured to:

display a second virtual object superimposed on the first real object to provide a notification related to the first industrial automation equipment; and

display the first virtual object in response to receiving a user selection of the second virtual object.

5. The first computing device of claim 1, wherein the first computing device is configured to communicate the image data to the second computing device to enable the second computing device to determine identity of the first industrial automation equipment;

wherein the second computing device is configured to determine the relevant information is based at least in part on the identity of the first industrial automation equipment.

6. The first computing device of claim 1, wherein the display component is configured to:

display a second real object corresponding with a second industrial automation equipment in the portion of the facility based at least in part on the image data; and

display a second virtual object superimposed on at least a portion of the plurality of real objects based at least in part on a set of the relevant information related to the second industrial automation equipment.

7. The first computing device of claim 1, wherein the display component is configured to:

display a button virtual object superimposed on at least a portion of the plurality of real objects; and

display a second virtual object superimposed on at least a portion of the plurality of real object in response to user selection of the button virtual object, wherein the second virtual object is configured to present additional relevant information associated with one of the plurality of real objects.

8. The first computing device of claim 1, wherein the first computing device comprises a mobile computing device configured to be portable about the facility and the display component comprises a touch-sensitive display.

9. The first computing device of claim 1, wherein the first computing device is configured to receive the relevant information from the first industrial automation equipment.

10. A tangible, non-transitory, computer-readable medium configured to store instructions executable by a processor in a computing device, wherein the instructions comprise instructions to:

determine, using the processor, a location and an orientation of the computing device with respect to physical features in a facility, wherein the physical features comprise a first industrial automation equipment;

instruct, using the processor, the computing device to display a plurality of real objects configured to visually represent one or more of the physical features, wherein the plurality of real objects comprises a first real object when the computing device is orientated toward and located proximate to the first industrial automation equipment, wherein the first real object is configured to visually represent the first industrial automation equipment;

instruct, using the processor, the computing device to display a first virtual object such that the first virtual object is superimposed on at least a portion of the plurality of real objects, wherein the first virtual object is configured to visually indicate relevant information associated with the first industrial automation equipment and is displayed adjacent to the first real object.

11. The computer-readable medium of claim 10, wherein the instructions to determine the location and the orientation comprise instructions to:

determine identities of the one or more of the physical features based at least in part on image data captured by the computing device;

determine physical dimensions of the one or more of the physical features based at least in part on the identities of the one or more of the physical features;

determine the location and the orientation of the computing device based at least in part on depiction of the physical dimensions of the one or more of the physical features in the image data.

12. The computer-readable medium of claim 11, comprising instructions to receive the physical dimensions of the one or more of the physical features from the industrial automation equipment.

13. The computer-readable medium of claim 10, comprising instructions to receive image data from another computing device or the industrial automation equipment based at least in part on the location and the orientation of the computing device, wherein the plurality of real objects is displayed based at least in part on the image data.

14. The computer-readable medium of claim 10, wherein the instructions to display the first virtual object comprise instructions to display the first virtual object on the display component when the first computing device is oriented toward the first industrial automation equipment and is located within a certain distance from the first industrial automation equipment.

15. The computer-readable medium of claim 10, wherein the plurality of real objects comprises a second real object when the computing device is orientated toward and located proximate to the second industrial automation equipment;

and

comprising instructions to display, using the processor, a second virtual object on the display component, wherein the second virtual object is superimposed on at least a portion of the plurality of real objects, and wherein the second virtual object presents a set of relevant information related to the second industrial automation equipment and is displayed adjacent to the second real object.

16. The computer-readable medium of claim 10, comprising instructions to perform a three-dimensional scan to capture a three-dimensional visual representation of the one or more physical features;

wherein the plurality of real objects is displayed based at least in part on the three-dimensional visual representation.

17. The computer-readable medium of claim 10, wherein the real objects are configured to visually represent physical features obstructed from view at the location and the orientation.

18. A method, comprising:

displaying, using a display component of a computing device in an industrial automation system, a real object that visually represents industrial automation equipment of the industrial automation system based at least in part on image data capturing the industrial automation equipment;

displaying, using the display component, a virtual object that visually indicates one or more operational parameters of the industrial automation equipment; and

determining, using a processor of the computing device, one or more procedure instructions, assessment information, or both based at least in part on the real object and the virtual object, wherein the one or more procedure instructions are configured to be executed to perform an operation in the industrial automation system and the assessment information comprises one or more recommendations to adjust the industrial automation system.

19. The method of claim 18, wherein determining the one or more procedure instructions, the assessment information, or both comprises:

transmitting, using a communication component of the computing device, the image data to another computing device associated with a user physically remote from a facility that includes the industrial automation equipment; and

receiving, using the communication component, the procedure instructions, the assessment information, or both from the another computing device.

20. The method of claim 18, wherein displaying the virtual object comprises displaying the virtual object to indicate potential replacement parts to use in the industrial automation equipment when the industrial automation equipment has an obsolete status, wherein the assessment information comprises the obsolete status.