Audio recognition-based industrial automation control

• US 10,719,066 B2
• Filed: 08/29/2018
• Issued: 07/21/2020
• Est. Priority Date: 08/29/2018
• Status: Active Grant

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Inventors: Nguyen, Thong, Schmirler, Paul David, Sanchez, Alvaro, Filtz, Darrell Steven, Litzau, Steven James
• Primary Examiner(s): Patel, Ramesh B
• Application Number:
  US16/116,605
• Publication Number:
  US 20200073367A1
• Time in Patent Office: 692 Days
• Field of Search: None
• CPC Class Codes: G05B 19/4183 characterised by data acquisition, e.g. workpiece identification
• CPC Class Codes: G05B 19/4184 characterised by fault tolerance, reliability of production system
• CPC Class Codes: G05B 2219/35458 Control command embedded in video, audio stream, signal
• CPC Class Codes: G05B 2219/37337 Noise, acoustic emission, sound
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content

Abstract

A system for performing industrial automation control may include an audio device that receives audio data from an element in an industrial automation system. The audio device may determine orientation data based on the audio data. In addition, the audio device may determine an automation command to control a machine in the industrial automation system based on the audio data and the orientation data. After determining the automation command, the audio device may implement a first control action for the machine based at least in part on the automation command, where the first control action causes the machine to adjust an operation.

Claims

1. A method for performing industrial automation control in an industrial automation system, comprising:

detecting, via an audio device, audio data by a user in the industrial automation system;

determining, via the audio device, orientation data configured to indicate a directionality of the user based at least in part on a strength of the audio data;

determining, via the audio device, an automation command that corresponds to a first control action configured to control one or more operations of a machine in the industrial automation system based on the audio data and the orientation data; and

implementing, via the audio device, the first control action for the machine based at least in part on the automation command.

2. The method of claim 1, comprising determining, via the audio device, the orientation data based at least in part on the strength of the audio data relative to additional strength of additional audio data.

3. The method of claim 2, comprising determining the machine associated with the automation command based on a relationship between the audio data and the orientation data.

4. The method of claim 1, comprising:

receiving, via the audio device, a secondary input configured to verify the automation command; and

determining, via the audio device, a confidence value for the automation command based on the audio data and the secondary input.

5. The method of claim 1, comprising:

determining, via the audio device, a source location of the audio data; and

determining, via the audio device, the machine associated with the automation command based on the source location.

6. The method of claim 1, wherein the automation command is determined based on at least one characteristic associated with the audio data, wherein the at least one characteristic comprises frequency of the audio data, of the audio data, volume of the audio data, intensity of the audio data, or any combination thereof.

7. The method of claim 1, comprising:

determining, via the audio device, a confidence value of the automation command based on the orientation data; and

implementing, via the audio device, the first control action based at least in part on the confidence value.

8. The method of claim 1, comprising determining the automation command configured based on a plurality of automation commands stored in a database and a plurality of audio datasets associated with the plurality of automation commands stored in the database, wherein determining the automation command comprises matching the audio data to one of the plurality of audio datasets based on a first set of characteristics in common between the audio data and the one of the plurality of audio datasets.

9. A system for performing industrial automation control, comprising:

an audio device, configured to;

receive audio data of an element in an industrial automation system;

determine orientation data configured to indicate a directionality of the element based on a strength of the audio data;

determine an automation command that corresponds to a first control action configured to control one or more operations of a machine in the industrial automation system based on the audio data, the orientation data, a plurality of automation commands stored in a database, and a plurality of audio datasets associated with the plurality of automation commands stored in the database; and

send the first control action to the machine based at least in part on the automation command, wherein the first control action is configured to cause the machine to adjust a respective operation of the one or more operations.

10. The system of claim 9, wherein the element is a user in the industrial automation system, and wherein the audio data comprises an audible command configured to instruct the machine to perform the respective operation.

11. The system of claim 10, wherein the audible command comprises at least two recitations of a word or phrase.

12. The system of claim 9, wherein the element comprises the machine in the industrial automation system, and wherein the audio data corresponds to an expected noise profile for the machine.

13. The system of claim 12, wherein the audio device is configured to:

determine a deviation of the audio data from the expected noise profile associated with the machine; and

determine whether the machine has a defect based on the deviation.

14. The system of claim 9, comprising an additional audio device, wherein the audio device is configured to determine a source location of the audio data based at least in part on the audio data and an additional audio data detected by the additional audio device.

15. The system of claim 14, wherein the audio device is configured to determine the first control action based on a relationship between the audio data and secondary input data configured to confirm the first control action.

16. The system of claim 9, wherein the audio device comprises a plurality of microphones and a processor, and wherein the processor is configured to:

receive detected audio strength data from each of the plurality of microphones, wherein the detected audio strength data comprises the strength of the audio data;

determine the orientation data at least in part by comparing the strength of the audio data from a first microphone of the plurality of microphones to an additional strength of the audio data detected by a second microphone of the plurality of microphones; and

determine the automation command based on the audio data, the detected audio strength data, the directionality of the element, the plurality of automation commands stored in the database, and the plurality of audio datasets associated with the plurality of automation commands stored in the database.

17. A tangible, non-transitory computer-readable medium configured to store instructions executable by a processor of an electronic device that, when executed by the processor, cause the processor to:

detect audio data associated with a first machine or user in an industrial automation system;

determine orientation data configured to indicate a directionality of the first machine or of the user based on the audio data;

determine an automation command that corresponds to a first control action configured to control one or more operations of a second machine in the industrial automation system based on the audio data and the orientation data; and

send the first control action to the second machine based at least in part on the automation command, wherein the first control action is configured to cause the second machine to adjust a respective operation of the one or more operations.

18. The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed by the processor, cause the processor to:

receive a first secondary input data from a sensor system associated with a presence of the user in the industrial automation system; and

determine the first control action to correspond to an instruction to operate the second machine into a powered-down state in response to the first secondary input data not matching expected secondary input data associated with the audio data.

19. The non-transitory computer-readable medium of claim 17, comprising instructions, when executed by the processor, cause the processor to:

receive an additional audio data from an additional audio device;

determine a source location of the audio data based on one or more triangulation schemes and the additional audio data; and

determine the automation command based on the source location.

20. The non-transitory computer-readable medium of claim 17, comprising instructions, when executed by the processor, cause the processor to:

receive an expected noise profile corresponding to an expected operation of the first machine;

determine a deviation of the audio data from the expected noise profile; and

determine the automation command based on the deviation.