Multimodal search input for an industrial search platform

• US 10,318,570 B2
• Filed: 08/18/2016
• Issued: 06/11/2019
• Est. Priority Date: 08/18/2016
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Billi-Duran, Sharon M., Como, Christopher W., Gray, Edward A., Reissner, Kyle K., Walter, Jonathan D., Singhai, Mohit, Reichard, Douglas J., Sandler, Scott N., Bliss, Ronald E., Pantaleano, Michael J., Cahalane, Ryan, Korpela, Jessica L., McCleave, Jr., Bruce T.
• Primary Examiner(s): Phan, Tuankhanh D
• Application Number:
  US15/240,161
• Publication Number:
  US 20180052835A1
• Time in Patent Office: 1,027 Days
• Field of Search: None
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location

Abstract

An industrial search and diagnosis system discovers available data items across heterogeneous data platforms that make up an industrial enterprise and indexes the data items in a unified searchable namespace. The system also indexes non-textual or multimedia input, including image data, video data, and audio data. This multimedia data can be indexed by the system in association with the relevant industrial systems (e.g., machines, automation systems, devices, production lines, etc.) so that the system can provide the multimedia data to the user when the relevant industrial systems are searched or invoked. The system also accepts and processes multimedia data from as search criteria or for diagnostic analysis. In an example analysis, users can submit images, audio, or video of an industrial machine or automation system, and the diagnosis system can identify an operational abnormality based on comparative analysis performed on the multimedia information.

Claims

1. A system for facilitating diagnosis of industrial performance issues, comprising:

a memory that stores executable components;

a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising;

a transform component configured to

receive multimedia data captured by a client device,

correlate location information representing a location of the client device at a time that the multimedia data was received with respective locations of automation systems recorded in a federated data model to yield a correlation result,

determine an automation system, of the automation systems, represented by the multimedia data based on the correlation result, and

tag the multimedia data with contextual information identifying the automation system represented by the multimedia data;

an indexing component configured to index the multimedia data in a federated data model of an industrial enterprise in association with the automation system based on the contextual information;

an analysis component configured to identify a performance issue associated with the automation system based on an analysis of the multimedia data; and

a notification component configured to send notification data to one or more recipient client devices in response to identification of the performance issue, the notification data including at least one of a guidance image or a guidance video.

2. The system of claim 1, wherein

the multimedia data is at least one of an image or a video, and

the transform component is configured to generate at least a portion of the contextual information based on an analysis of one or more identifying characteristics contained in the image or the video.

3. The system of claim 2, wherein the one or more identifying characteristics comprise at least one of nameplate information extracted from the image or video or a visual property of the automation system identified in the image or video.

4. The system of claim 1, wherein

the multimedia data comprises an audio maintenance recording recorded on the client device and submitted to the transform component, and

the notification component is further configured to deliver the maintenance recording to another client device in response to a determination that the other client device has moved within a defined distance from the automation system.

5. The system of claim 1, wherein

the indexing component is configured to classify the multimedia data as reference data in a selected one of multiple categories defined in the federated data model, and

the multiple categories comprise at least a first category representing normal operation of the automation system and one or more second categories representing respective one or more abnormal conditions of the automation system.

6. The system of claim 5, wherein

the multimedia data is first multimedia data,

the transform component is further configured to receive second multimedia data representing the automation system from the client device or another client device, and

the analysis component is configured to perform comparative analysis on the second multimedia data relative to the reference data, and to identify the performance issue associated with the automation system based on a result of the comparative analysis.

7. The system of claim 6, wherein the comparative analysis comprises

inferring, based on analysis of the first multimedia data and the second multimedia data, at least one of vibration rates of a machine component or product throughput rates of the automation system for each of the first multimedia data and the second multimedia data, and

generating the result based on a comparison of the vibration rates or the product throughput rates for the first multimedia data and the second multimedia data.

8. The system of claim 6, wherein the notification component is configured to select the one or more recipient client devices based on one or more notification rules that define one or more notification recipients that are associated with the automation system.

9. The system of claim 1, wherein

the multimedia data comprises an audio maintenance recording recorded on the client device and submitted to the transform component,

the indexing component is configured to associate the maintenance recording with a human-machine interface terminal associated with the automation system identified by the transform component,

the executable components further comprise a device interface component configured to render an interactive link on the human-machine interface terminal indicating availability of the maintenance recording, and

the notification component is further configured to, in response to selection of the interactive link via the human-machine interface terminal, deliver the maintenance recording to the human-machine interface terminal.

10. The system of claim 5, wherein the one or more abnormal conditions comprise at least one of a leaking pump, a slipping belt, or excessive machine vibration.

11. The system of claim 6, wherein

the reference data has associated therewith metadata identifying a portion of an image or video represented by the second multimedia data that is to be analyzed for presence of a visual characteristic indicative of the performance issue, and

the analysis component is configured to perform the comparative analysis on the portion of the image or video indicated by the metadata.

12. A method for identifying industrial performance issues based on multimedia information, comprising:

receiving, by a system comprising at least one processor, multimedia data recorded by a client device;

receiving, by the system, location information identifying a location of the client device at a time that the multimedia data was received;

correlating, by the system, the location information with respective locations of automation systems recorded in a federated data model;

determining, by the system, an automation system, of the automation systems, represented by the multimedia data based on a result of the correlating;

tagging, by the system, the multimedia data with contextual information identifying the automation system represented by the multimedia data;

indexing, by the system and based on the contextual information, the multimedia data in a federated data model of an industrial enterprise in association with the automation system;

identifying, by the system based on an analysis performed on the multimedia data, a performance issue associated with the automation system; and

in response to the identifying, sending, by the system, notification data to one or more recipient client devices, the notification data including at least one of a guidance image or a guidance video.

13. The method of claim 12, wherein

the receiving comprises receiving at least one of an image or a video of the automation system, and

the tagging comprises generating at least a portion of the contextual information based on an analysis of one or more identifying characteristics contained in the image or the video.

14. The method of claim 13, wherein the tagging further comprises identifying, as the one or more identifying characteristics, at least one of nameplate information contained within the image or video or a visual property of the automation system recorded in the image or video.

15. The method of claim 12, wherein the receiving comprises receiving, as the multimedia data, an audio maintenance recording recorded on the client device, and the method further comprises:

tracking, by the system, a location of another client device associated with a user role defined as being responsible for the automation system; and

in response to determining, based on the tracking, that the location is within a defined distance from the automation system, sending, by the system, the audio maintenance recording to the other client device.

16. The method of claim 12, wherein the indexing comprises classifying the multimedia data in the federated data model as reference data representing one of normal operation of the automation system or an abnormal condition of the automation system.

17. The method of claim 16, wherein the multimedia data is first multimedia data, and the identifying the performance issue comprises:

receiving, by the system, second multimedia data representing the automation system from the client device or another client device;

performing, by the system, a comparative analysis that compares the second multimedia data with the reference data; and

identifying, by the system based on a result of the comparative analysis, the performance issue associated with the automation system.

18. A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a system comprising a processor to perform operations, the operations comprising:

receiving multimedia data recorded by a client device;

receiving location information that identifies a location of the client device at a time that the multimedia data was received;

correlating the location information with respective locations of automation systems recorded in a federated data model;

identifying an automation system, of the automation systems, represented by the multimedia data based on a result of the correlating;

appending contextual information to the multimedia data, the contextual information identifying the automation system represented by the multimedia data;

indexing, based on the contextual information, the multimedia data in a federated data model of an industrial enterprise in association with the automation system;

identifying, based on an analysis performed on the multimedia data, a performance issue associated with the automation system; and

in response to the identifying the performance issue, sending notification data to one or more recipient client devices, the notification data including at least one of a guidance image or a guidance video.

19. The non-transitory computer-readable medium of claim 18, wherein

the receiving the multimedia data comprises receiving at least one of an image or a video of the automation system, and

the appending comprises generating at least a portion of the contextual information based on an analysis of one or more identifying characteristics contained in the image or the video.

20. The non-transitory computer-readable medium of claim 19, wherein the one or more identifying characteristics comprise at least one of nameplate information extracted from the at least one of the image or the video or a visual property of the automation system identified in the at least one of the image or the video.