Systems and methods for presenting an augmented reality

• US 10,950,051 B2
• Filed: 06/30/2015
• Issued: 03/16/2021
• Est. Priority Date: 03/27/2015
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Michalscheck, Jimi R., Korpela, Jessica L., Reissner, Kyle K., Vasko, David A., Fordenwalt, Matthew W., Jauquet, John J., Ericsson, Matthew R., Wilber, Andrew, Michalscheck, Kelly A.
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): Vu, Khoa
• Application Number:
  US14/788,557
• Publication Number:
  US 20160284128A1
• Time in Patent Office: 2,086 Days
• CPC Class Codes: G05B 19/409 characterised by using manual data input [MDI] or by using control panel, e.g. controlling functions with the panel; characterised by control panel details or by setting parameters G05B19/408, G05B19/4093 take precedence
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05B 2219/33286 Test, simulation analysator
• CPC Class Codes: G05B 2219/36167 Use camera of handheld device, pda, pendant, head mounted display
• CPC Class Codes: G05B 23/0216 Human interface functionality, e.g. monitoring system providing help to the user in the selection of tests or in its configuration
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A first computing device in an industrial automation system may include an image sensor that captures a visual representation of a portion of a facility including a first industrial automation equipment as image data and a communication component that communicatively couples the first computing device to a second computing device via a communication network, in which the first computing device receives relevant information related to the first industrial automation equipment from the second computing device. The first computing may include a display component that displays a plurality of real objects that visually represent physical features in the portion of the facility based at least in part on the image data, in which the plurality of real objects include a first real object that visual represents physical features of the first industrial automation equipment; and that displays a first virtual object superimposed on at least a portion of the plurality of real objects based at least in part on the relevant information.

Claims

1. A first computing device in an industrial automation system comprising:

an image sensor configured to capture one or more images of a portion of a facility as image data;

a communication component configured to communicatively couple the first computing device to a second computing device via a communication network to enable the first computing device to receive relevant information related to a first industrial automation equipment from the second computing device, wherein the relevant information comprises one or more operational parameters transmitted by one or more sensors associated with the first industrial automation equipment; and

a display component configured to;

display a plurality of real objects that depicts physical features in the portion of the facility based at least in part on the image data, wherein the plurality of real objects comprises an obstructing real object depicting a physical obstruction disposed between the image sensor and the first industrial automation equipment in response to the image sensor being oriented toward the first industrial automation equipment, and wherein at least a portion of the first industrial automation equipment is blocked from view of the image sensor by the physical obstruction;

modify the obstructing real object into an outline of the physical obstruction to present a first virtual object that depicts the first industrial automation equipment within the outline, wherein the first virtual object is generated based at least in part on the operational parameters and is configured to reveal a representation of the first industrial automation equipment through the outline of the physical obstruction within the image data; and

display a second virtual object superimposed on at least a portion of the plurality of real objects, wherein the second virtual object is generated based at least in part on the relevant information in response to the first computing device being moved within a threshold distance from the first industrial automation equipment and the image sensor being oriented toward the first industrial automation equipment, and wherein the second virtual object comprises at least a portion of the operational parameters.

2. The first computing device of claim 1, comprising a location sensor configured to determine a location of the first computing device with respect to a set of the physical features of the facility, wherein the display component is configured to display the second virtual object superimposed on the at least a portion of the plurality of real objects in response to the first virtual object being displayed.

3. The first computing device of claim 1, wherein the display component is configured to display the second virtual object in response to receiving a user selection of the first virtual object.

4. The first computing device of claim 1, wherein the display component is configured to:

display a third virtual object superimposed on the first virtual object to provide a notification related to the first industrial automation equipment; and

display the second virtual object in response to receiving a user selection of the third virtual object.

5. The first computing device of claim 1, wherein the first computing device is configured to communicate the image data to the second computing device to enable the second computing device to determine an identity of the first industrial automation equipment blocked from view of the image sensor by the physical obstruction and to determine the relevant information based at least in part on the identity of the first industrial automation equipment.

6. The first computing device of claim 1, wherein the display component is configured to:

display a first real object depicting a second industrial automation equipment in the portion of the facility using the image data in response to the image sensor being oriented toward the second industrial automation equipment; and

display a third virtual object superimposed on at least a portion of the plurality of real objects based at least in part on a set of the relevant information related to the second industrial automation equipment in response to the first computing device being moved within the threshold distance from the second industrial automation equipment and the image sensor being oriented toward the second industrial automation equipment.

7. The first computing device of claim 1, wherein the display component is configured to:

display a button virtual object superimposed on at least a portion of the plurality of real objects; and

display a third virtual object superimposed on at least a portion of the plurality of real objects in response to user selection of the button virtual object, wherein the second virtual object is configured to present additional relevant information associated with one of the plurality of real objects.

8. The first computing device of claim 1, wherein the first computing device comprises a mobile computing device configured to be portable about the facility and the display component comprises a touch-sensitive display.

9. The first computing device of claim 1, wherein the first virtual object depicting the first industrial automation equipment through the physical obstruction is based at least in part on previously captured image data of the first industrial automation equipment.

10. The first computing device of claim 1, wherein the first industrial automation equipment comprises an internal component of power equipment, and wherein the physical obstruction comprises a housing of the power equipment.

11. The first computing device of claim 1, wherein the physical obstruction comprises a second industrial equipment.

12. The first computing device of claim 1, wherein the image data comprises image data captured by a three-dimensional scan of the industrial automation system.

13. A tangible, non-transitory, computer-readable medium storing instructions executable by at least one processor in a computing device, wherein the instructions comprise instructions to:

determine, using the at least one processor, a location and an orientation of the computing device with respect to physical features in a facility, wherein the physical features comprise a first industrial automation equipment that is at least partially blocked from view of an image sensor of the computing device at the location and the orientation by a physical obstruction disposed between the image sensor and the first industrial automation equipment, and wherein the location and the orientation are determined by;

determining identities of one or more of the physical features based at least in part on image data captured by the image sensor;

determining physical dimensions of the one or more of the physical features based at least in part on the identities of the one or more of the physical features; and

determining the location and the orientation of the computing device based at least in part on depiction of the physical dimensions of the one or more of the physical features in the image data;

instruct, using the at least one processor, the computing device to display a plurality of real objects based at least in part on the image data captured by the image sensor of the computing device, wherein the plurality of real objects comprises an obstructing real object depicting the physical obstruction in response to the image sensor of the computing device being oriented toward the first industrial automation equipment;

instruct, using the at least one processor, the computing device to modify the obstructing real object into an outline of the physical obstruction to present a first virtual object depicting a component of the first industrial automation equipment within the outline, wherein the first virtual object is generated based at least in part on operational parameter data received from the component and is configured to reveal a representation of the component through the outline of the physical obstruction within the image data; and

instruct, using the at least one processor, the computing device to display a second virtual object superimposed on at least a portion of the plurality of real objects, wherein the second virtual object is generated in response to the computing device being moved within a threshold distance from the first industrial automation equipment and the image sensor of the computing device being oriented toward the first industrial automation equipment, and wherein the second virtual object is configured to visually indicate the operational parameter data associated with the component of the first industrial automation equipment and is displayed adjacent to the first virtual object.

14. The computer-readable medium of claim 13, comprising instructions to receive the physical dimensions of the one or more of the physical features from the first industrial automation equipment.

15. The computer-readable medium of claim 13, comprising instructions to receive additional image data from another computing device or the first industrial automation equipment based at least in part on the location and the orientation of the computing device, wherein the plurality of real objects is displayed using the additional image data.

16. The computer-readable medium of claim 13, wherein the plurality of real objects comprises a second real object depicting second industrial automation equipment in response to the image sensor of the computing device being oriented toward the second industrial automation equipment;

and

comprising additional instructions to instruct, using the at least one processor, the computing device to display a third virtual object superimposed on at least a portion of the plurality of real objects in response to the computing device being moved within the threshold distance from the second industrial automation equipment and the image sensor of the computing device being oriented toward the second industrial automation equipment, wherein the third virtual object presents a set of relevant information related to the second industrial automation equipment and is displayed adjacent to the second real object.

17. A method, comprising:

displaying, using a display component of a computing device in an industrial automation system, an obstructing real object that depicts a physical obstruction disposed between industrial automation equipment and an image sensor of the computing device based at least in part on image data captured by the image sensor in response to the image sensor being oriented toward the industrial automation equipment, wherein at least a portion of the industrial automation equipment is blocked from view of the image sensor by the physical obstruction;

modifying, using the display component, the obstructing real object into an outline of the physical obstruction to present a first virtual object that depicts the industrial automation equipment within the outline, wherein the first virtual object is generated based at least in part on one or more operational parameters transmitted by one or more sensors associated with the industrial automation equipment, and wherein the real object is configured to reveal a representation of the industrial automation equipment through the outline of the physical obstruction within the image data;

displaying, using the display component, a second virtual object that visually indicates at least a portion of the operational parameters of the industrial automation equipment superimposed on at least a portion of the first virtual object, wherein the second virtual object is generated in response to the computing device being moved within a threshold distance from the industrial automation equipment and the image sensor of the computing device being oriented toward the industrial automation equipment; and

determining, using a processor of the computing device, one or more procedure instructions, assessment information, or both based at least in part on the real object and the virtual object, wherein the one or more procedure instructions are configured to be executed to perform an operation in the industrial automation system and the assessment information comprises one or more recommendations to adjust the industrial automation system.

18. The method of claim 17, wherein determining the one or more procedure instructions, the assessment information, or both comprises:

transmitting, using a communication component of the computing device, the image data to another computing device associated with a user physically remote from a facility that includes the industrial automation equipment; and

receiving, using the communication component, the procedure instructions, the assessment information, or both from the another computing device.

19. The method of claim 17, wherein displaying the second virtual object comprises displaying the second virtual object to indicate potential replacement parts to use in the industrial automation equipment in response to the industrial automation equipment having an obsolete status, and wherein the assessment information comprises the obsolete status.

20. The method of claim 17, comprising determining, using the processor, a location and an orientation of the computing device by:

determining, using the processor, identities of one or more physical features including the industrial automation equipment based at least in part on the image data captured by the image sensor of the computing device;

determining, using the processor, physical dimensions of the one or more of the physical features based at least in part on the identities of the one or more of the physical features; and

determining, using the processor, the location and the orientation of the computing device based at least in part on depiction of the physical dimensions of the one or more of the physical features in the image data.