User interface sound emanation activity classification

• US 10,503,467 B2
• Filed: 07/13/2017
• Issued: 12/10/2019
• Est. Priority Date: 07/13/2017
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Abuelsaad, Tamer E., Boss, Gregory J., Moore, Jr., John E., Rendahl, Randy A.
• Primary Examiner(s): Vu, Kieu D
• Assistant Examiner(s): Calderon, IV, Alvaro R
• Application Number:
  US15/648,628
• Publication Number:
  US 20190018646A1
• Time in Patent Office: 880 Days
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 8/61 Installation
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

Methods, computer program products, and systems are presented. The method computer program products, and systems can include, for instance: obtaining an audio input, the audio input representing sound emanating from a key press based user interface; generating a context pattern based on the audio input; performing classification of the context pattern to classify the context pattern as belonging to a signature pattern classification, wherein the signature pattern classification specifies a user activity; and providing an output based on the performing classification.

Claims

1. A computer program product comprising:

a computer readable storage medium readable by one or more processing unit and storing instructions for execution by one or more processor for performing a method comprising;

obtaining an audio input, the audio input representing sound emanating from a key press based user interface;

generating a context pattern based on the audio input;

performing classification of the context pattern to classify the context pattern as belonging to a signature pattern classification, wherein the signature pattern classification specifies a user activity; and

providing an output based on the performing classification, wherein the performing classification results in classification failure, wherein the method includes prompting a user to enter activity information in response to the classification failure, and wherein the prompting includes providing user interface functionality that prompts the user to enter activity specifying information that specifies an activity engaged in by the user resulting in the generating of the context pattern, and wherein the method includes, using data received from the user in response to the prompting, registering the context pattern as a new signature pattern for comparison to subsequent context patterns.

2. The computer program product of claim 1, wherein the obtaining, the generating, and the performing classification is performed for each of first through sixth audio inputs, wherein the performing classification of a context pattern associated to the first audio input results in a classification according to a Web Browsing signature pattern classification, wherein the performing classification of a context pattern associated to the second audio input results in a classification according to a Social Media signature pattern classification, wherein the performing classification of a context pattern associated to the third audio input results in a classification according to a Mass Messaging signature pattern classification, wherein the performing classification of a context pattern associated to the fourth audio input results in a classification according to a Gaming signature pattern classification, wherein the performing classification of a context pattern associated to the fifth audio input results in a classification according to a Word Processing signature pattern classification, wherein the performing classification of a context pattern associated to the sixth audio input results in a classification according to a Specific String signature pattern classification.

3. The computer program product of claim 1, wherein the obtaining, the generating, and the performing classification is performed for each of first and second audio inputs, wherein the performing classification of a context pattern associated to the first audio input results in a classification according to a Web Browsing signature pattern classification, wherein the performing classification of a context pattern associated to the second audio input results in a classification according to a Gaming signature pattern classification.

4. A system comprising:

a memory;

at least one processor in communication with the memory; and

program instructions executable by one or more processor via the memory to perform a method comprising;

obtaining an audio input, the audio input representing sound emanating from a key press based user interface;

generating a context pattern based on the audio input;

performing classification of the context pattern to classify the context pattern as belonging to a signature pattern classification, wherein the signature pattern classification specifies a user activity; and

providing an output based on the performing classification, wherein the performing classification results in classification failure, wherein the method includes prompting a user to enter activity information in response to the classification failure, and wherein the prompting includes providing user interface functionality that prompts the user to enter activity specifying information that specifies an activity engaged in by the user resulting in the generating of the context pattern, and wherein the method includes, using data received from the user in response to the prompting, registering the context pattern as a new signature pattern for comparison to subsequent context patterns.

5. The system of claim 4, wherein the obtaining, the generating, and the performing classification is performed for each of first through sixth audio inputs, wherein the performing classification of a context pattern associated to the first audio input results in a classification according to a Web Browsing signature pattern classification, wherein the performing classification of a context pattern associated to the second audio input results in a classification according to a Social Media signature pattern classification, wherein the performing classification of a context pattern associated to the third audio input results in a classification according to a Mass Messaging signature pattern classification, wherein the performing classification of a context pattern associated to the fourth audio input results in a classification according to a Gaming signature pattern classification, wherein the performing classification of a context pattern associated to the fifth audio input results in a classification according to a Word Processing signature pattern classification, wherein the performing classification of a context pattern associated to the sixth audio input results in a classification according to a Specific String signature pattern classification.

6. The system of claim 4, wherein the obtaining, the generating, and the performing classification is performed for each of first and second audio inputs, wherein the performing classification of a context pattern associated to the first audio input results in a classification according to a Web Browsing signature pattern classification, wherein the performing classification of a context pattern associated to the second audio input results in a classification according to a Gaming signature pattern classification.

7. A computer program product comprising:

a computer readable storage medium readable by one or more processing unit and storing instructions for execution by one or more processor for performing a method comprising;

obtaining an audio input, the audio input representing sound emanating from key presses of a key press based user interface;

generating a context pattern based on the audio input;

performing classification of the context pattern to classify the context pattern as belonging to a signature pattern classification, wherein the signature pattern classification specifies a user activity, the user activity involving use of keys of the key press based user interface; and

providing an output based on the performing classification, wherein the performing classification results in classification failure, wherein the method includes prompting a user to enter activity information in response to the classification failure, and wherein the prompting includes providing user interface functionality that prompts the user to enter activity specifying information that specifies an activity engaged in by the user resulting in the generating of the context pattern, and wherein the method includes, using data received from the user in response to the prompting, registering the context pattern as a new signature pattern for comparison to subsequent context patterns.

8. The computer program product of claim 7, wherein the obtaining, the generating, and the performing classification is performed for each of first through sixth audio inputs, wherein the performing classification of a context pattern associated to the first audio input results in a classification according to a Web Browsing signature pattern classification, wherein the performing classification of a context pattern associated to the second audio input results in a classification according to a Social Media signature pattern classification, wherein the performing classification of a context pattern associated to the third audio input results in a classification according to a Mass Messaging signature pattern classification, wherein the performing classification of a context pattern associated to the fourth audio input results in a classification according to a Gaming signature pattern classification, wherein the performing classification of a context pattern associated to the fifth audio input results in a classification according to a Word Processing signature pattern classification, wherein the performing classification of a context pattern associated to the sixth audio input results in a classification according to a Specific String signature pattern classification, wherein the Web Browsing signature pattern classification specifies that the user is using key presses of the key press based user interface to perform Web Browsing, wherein the Social Media signature pattern classification specifies that the user is using key presses of the key press based user interface to use Social Media, wherein the Mass Messaging signature pattern classification specifies that the user is using key presses of the key press based user interface to perform Mass Messaging, wherein the Gaming signature pattern classification specifies that the user is using key presses of the key press based user interface to engage in Gaming, wherein the Word Processing signature pattern classification specifies that the user is using key presses of the key press based user interface to perform Word Processing.

9. The computer program product of claim 7, wherein the signature pattern classification is a classification defined by a certain character string.

10. The computer program product of claim 7, wherein the method includes providing functionality that allows an administrator user to enter a character string, registering a first context pattern as a first signature pattern associated to the character string, and wherein the performing classification includes matching an incoming context pattern to the first signature pattern.

11. The computer program product of claim 7, wherein the providing an output includes providing a notification to an external system.

12. The computer program product of claim 7, wherein the providing an output includes providing an output to configure a network hardware component.

13. The computer program product of claim 7, wherein the providing an output includes providing an output to initiate installation of software on a computer device.

14. The computer program product of claim 7, wherein the context pattern includes key press sequence information and timing information.

15. The computer program product of claim 7, wherein the obtaining includes obtaining the audio input from a computer device, wherein responsively to the performing classification failing to classify the context pattern, there is provided a prompt on the computer device prompting a user of the computer device to enter labeling information that specifies an activity associated with the context pattern.

16. The computer program product of claim 7, wherein the method includes providing functionality that allows an administrator user to perform training use of an administrator user key press based user interface during a training period, providing a certain audio input representing sound emanating from the administrator user key press based user interface during the training period, providing functionality that allows the administrator user to enter labeling information that specifies an activity associated to the training use, registering a first context pattern generated using the certain audio input as a first signature pattern labeled with the labeling information stored in a data repository, and matching an incoming context pattern to the first signature pattern.

17. The computer program product of claim 7, wherein the signature pattern classification is a Web Browsing classification.

18. The computer program product of claim 7, wherein the signature pattern classification is a Gaming classification.

19. The computer program product of claim 7, wherein the signature pattern classification is a Word Processing classification.