METHODS AND SYSTEMS FOR DATA COLLECTION, LEARNING, AND STREAMING OF MACHINE SIGNALS FOR PART IDENTIFICATION AND OPERATING CHARACTERISTICS DETERMINATION USING THE INDUSTRIAL INTERNET OF THINGS

• US 20200133254A1
• Filed: 11/14/2019
• Published: 04/30/2020
• Est. Priority Date: 05/07/2018
• Status: Active Application

• Current Assignee: Strong Force IoT Portfolio 2016, LLC (Strong Force Intellectual Capital, LLC)
• Original Assignee: Strong Force IoT Portfolio 2016, LLC (Strong Force Intellectual Capital, LLC)
• Inventors: Cella, Charles Howard, Duffy, JR., Gerald William, McGuckin, Jeffrey P., Desai, Mehul
• Application Number:
  US16/684,135
• Publication Number:
  US 20200133254A1
• Time in Patent Office: Days
• CPC Class Codes: G01M 13/028 Acoustic or vibration analysis
• CPC Class Codes: G01M 13/045 Acoustic or vibration analysis
• CPC Class Codes: G05B 13/028 using expert systems only
• CPC Class Codes: G05B 19/406 characterised by monitoring or safety G05B19/19 takes precedence
• CPC Class Codes: G05B 19/4183 characterised by data acquisition, e.g. workpiece identification
• CPC Class Codes: G05B 19/4184 characterised by fault tolerance, reliability of production system
• CPC Class Codes: G05B 19/41845 characterised by system universality, reconfigurability, modularity
• CPC Class Codes: G05B 19/4185 characterised by the network communication
• CPC Class Codes: G05B 19/41865 characterised by job scheduling, process planning, material flow
• CPC Class Codes: G05B 19/41875 characterised by quality surveillance of production
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05B 2219/32287 Medical, chemical, biological laboratory
• CPC Class Codes: G05B 2219/35001 Data input, data handling, programming, monitoring of nc
• CPC Class Codes: G05B 2219/37337 Noise, acoustic emission, sound
• CPC Class Codes: G05B 2219/37351 Detect vibration, ultrasound
• CPC Class Codes: G05B 2219/37434 Measuring vibration of machine or workpiece or tool
• CPC Class Codes: G05B 2219/40115 Translate goal to task program, use of expert system
• CPC Class Codes: G05B 2219/45004 Mining
• CPC Class Codes: G05B 2219/45129 Boring, drilling
• CPC Class Codes: G05B 23/0221 Preprocessing measurements, e.g. data collection rate adjustment; Standardization of measurements; Time series or signal analysis, e.g. frequency analysis or wavelets; Trustworthiness of measurements; Indexes therefor; Measurements using easily measured parameters to estimate parameters difficult to measure; Virtual sensor creation; De-noising; Sensor fusion; Unconventional preprocessing inherently present in specific fault detection methods like PCA-based methods
• CPC Class Codes: G05B 23/0229 : knowledge based, e.g. expert systems; genetic algorithms
• CPC Class Codes: G05B 23/024 : Quantitative history assessment, e.g. mathematical relationships between available data; Functions therefor; Principal component analysis [PCA]; Partial least square [PLS]; Statistical classifiers, e.g. Bayesian networks, linear regression or correlation analysis; Neural networks
• CPC Class Codes: G05B 23/0264 : Control of logging system, e.g. decision on which data to store; time-stamping measurements
• CPC Class Codes: G05B 23/0283 : Predictive maintenance, e.g. involving the monitoring of a system and, based on the monitoring results, taking decisions on the maintenance schedule of the monitored system; Estimating remaining useful life [RUL] preventive maintenance, i.e. planning maintenance according to the available resources without monitoring the system G06Q10/06
• CPC Class Codes: G05B 23/0286 : Modifications to the monitored process, e.g. stopping operation or adapting control
• CPC Class Codes: G05B 23/0289 : Reconfiguration to prevent failure, e.g. usually as a reaction to incipient failure detection
• CPC Class Codes: G05B 23/0291 : Switching into safety or degraded mode, e.g. protection and supervision after failure
• CPC Class Codes: G05B 23/0294 : Optimizing process, e.g. process efficiency, product quality
• CPC Class Codes: G05B 23/0297 : Reconfiguration of monitoring system, e.g. use of virtual sensors; change monitoring method as a response to monitoring results
• CPC Class Codes: G06F 16/2379 : Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/24 : Querying
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/2178 : based on feedback of a supervisor
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/006 : based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/02 : Neural networks
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/047 : Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 : Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 3/126 : Evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06N 5/046 : Forward inferencing; Production systems
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/20 : Administration of product repair or maintenance
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/20084 : Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30164 : Workpiece; Machine component
• CPC Class Codes: G06T 2207/30168 : Image quality inspection
• CPC Class Codes: G06T 7/0004 : Industrial image inspection
• CPC Class Codes: H04B 17/309 : Measuring or estimating channel quality parameters
• CPC Class Codes: H04B 17/318 : Received signal strength
• CPC Class Codes: H04L 1/0002 : by adapting the transmission rate
• CPC Class Codes: H04L 1/0041 : Arrangements at the transmitter end
• CPC Class Codes: H04L 1/18 : Automatic repetition systems, e.g. Van Duuren systems
• CPC Class Codes: H04L 5/0064 : Rate requirement of the data, e.g. scalable bandwidth, data priority
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: Y02P 80/10 : Efficient use of energy, e.g. using compressed air or pressurized fluid as energy carrier
• CPC Class Codes: Y02P 90/02 : Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
• CPC Class Codes: Y02P 90/80 : Management or planning
• CPC Class Codes: Y04S 40/18 : Network protocols supporting networked applications, e.g. including control of end-device applications over a network

Abstract

An industrial machine predictive maintenance system may include an industrial machine data analysis facility that generates streams of industrial machine health monitoring data by applying machine learning to data representative of conditions of portions of industrial machines received via a data collection network. The system may perform a method of image capture of a portion of an industrial machine in which an image capture template is provided and aligned via augmented reality with a live image in order to update a procedure for performing a service that implements a predicted maintenance action on an industrial machine. The system may perform a method of machine learning-based part recognition in which a captured image is analyzed and used to adapt a target part template, image analysis rules, or part recognition. The system may detect operating characteristics of an industrial machine via a machine learning aspect trained based on image data sets.

Claims

1. A method of image capture of a portion of an industrial machine, comprising:

updating a procedure for performing a service that implements a predicted maintenance action on an industrial machine, the updating responsive to a trigger condition for capturing an image of a portion of the industrial machine being met;

providing an image capture template in an electronic display overlaying a live image of a portion of the industrial machine to facilitate image capture;

applying augmented reality that indicates a degree of alignment of the live image with the template;

examining an image captured using the updated procedure with machine vision to determine at least one part of the machine present in the captured image; and

responsive to a result of the examination, operating an image capture reward facility to generate a reward for the captured image.

2. The method of claim 1, wherein the updating is responsive to a trigger condition that is based on analysis of industrial machine failure data such that the analysis suggests capturing an image that is not specified in the procedure prior to updating the procedure for performing the service.

3. The method of claim 1, wherein the updating is responsive to the procedure for performing the service being performed on an industrial machine that meets a predictive maintenance criterion associated with the portion of the industrial machine for which an image is to be captured.

4. The method of claim 1, wherein the trigger condition comprises a type of industrial machine associated with the industrial machine for which a service procedure is being performed and a duration of time since the portion of the industrial was captured in an image.

5. The method of claim 1, wherein the trigger condition relates to vibrations detected for at least a portion of the industrial machine, wherein an industrial machine service recommendation is produced according to a severity unit calculated for the detected vibrations.

6. The method of claim 5, wherein the severity unit is calculated for the detected vibrations of the industrial machine by determining a frequency of the detected vibrations, determining a segment of a multi-segment vibration frequency spectra that bounds the detected vibrations, and calculating the severity unit for the detected vibrations based on the determined segment.

7. The method of claim 6, wherein the segment of a multi-segment vibration frequency spectra that bounds the detected vibrations is determined by mapping the detected vibrations to one of a number of severity units based on the determined segment, wherein each of the severity units corresponds to a different range of the multi-segment vibration frequency spectra.

8. The method of claim 7, wherein the detected vibrations are mapped to a first severity unit when the frequency of the captured vibration corresponds to a below a low-end knee threshold-range of the multi-segment vibration frequency spectra, wherein the detected vibrations are mapped to a second severity unit when the frequency of the captured vibration corresponds to a mid-range of the multi-segment vibration frequency spectra, wherein the detected vibrations are mapped to a third severity unit when the frequency of the captured vibration corresponds to an above a high-end knee threshold-range of the multi-segment vibration frequency spectra.

9. A method of machine learning-based part recognition, comprising:

applying a target part imaging template to an image validating procedure that determines if an image captured meets an image capture validation criterion;

performing image analysis by processing a captured image with image analysis rules that facilitate detecting candidate parts of an industrial machine being present in an image;

recognizing one or more parts of the candidate parts as a part of the industrial machine based on similarity of a candidate part with images of parts of the industrial machine; and

adapting at least one of the target part template, the image analysis rules, and the part recognition based on feedback produced from machine learning of the recognized parts, thereby improving at least one of image capture, image analysis and part recognition.

10. The method of claim 9, wherein the images of parts of the industrial machine are retrieved from a smart RFID element disposed with the industrial machine.

11. The method of claim 9, wherein the images of parts of the industrial machine are retrieved from a network stored digital twin of a smart RFID element disposed with the industrial machine.

12. The method of claim 9, wherein the feedback produced by the machine learning facilitates updating a most recently captured image of the part in a smart RFID deployed with the industrial machine.

13. The method of claim 9, wherein some or all of the method is performed using a computer vision system.

14. The method of claim 13, wherein the computer vision system generates one or more image data sets using captured raw data, identifies one or more values corresponding to a portion of the industrial machine within a point of interest represented by the one or more image data sets, compares the one or more values to corresponding predicted values, generate a variance data set based on the comparison of the one or more values and the corresponding predicted values, detects an operating characteristic of the industrial machine based on the variance data, and generates data indicating the detection of the operating characteristic.

15. A system for detecting operating characteristics of an industrial machine, comprising:

at least one image data capture device that captures raw data of a point of interest of the industrial machine;

a computing device that generates one or more image data sets using the captured raw data, wherein a portion of the industrial machine associated with the point of interest is represented within the one or more image data sets;

an intelligent system that trains a machine learning aspect associated with the computing device based on the one or more image data sets by;

comparing one or more values from the one or more image data sets to corresponding predicted values;

generating a variance data set based on the comparison of the one or more values and the corresponding predicted values;

identifying an operating characteristic of the industrial machine based on the variance data;

determining whether the operating characteristic is within a tolerance based on whether the operating characteristic is greater than a threshold;

generating training data indicating the operating characteristic; and

training the machine learning aspect according to the training data.

16. The system of claim 15, wherein the intelligent system is a deep learning system.

17. The system of claim 15, wherein the intelligent system comprises one or more of a cognitive learning module, an artificial intelligence module, or a machine learning module.

18. The system of claim 15, wherein the training data includes image data associated with the industrial machine and non-image data associated with the industrial machine.

19. The system of claim 18, wherein the training data is stored in a knowledge base associated with the industrial machine.

20. The system of claim 19, wherein the knowledge base is updated based on output from the intelligent system.