SYSTEMS AND METHODS FOR DISCOVERING AUTOMATABLE TASKS

• US 20180113780A1
• Filed: 10/26/2017
• Published: 04/26/2018
• Est. Priority Date: 10/26/2016
• Status: Active Grant

• Current Assignee: Soroco Private Limited
• Original Assignee: Soroco Private Limited
• Inventors: Kim, Yoongu, Qadir, Abdul, Narayanaswamy, Arjun, Murty, Rohan Narayan, Barratt, Shane, Nychis, George Peter
• Granted Patent: US 10,831,450 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3423 where the assessed time is active or idle time
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 8/30 Creation or generation of source code
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06Q 10/06398 Performance of employee with respect to a job function
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display

Abstract

According to at least one aspect, a system for collecting computer usage information is provided. The system includes a hardware processor, a display coupled to the hardware processor to display a user interface, and a computer-readable storage medium storing processor-executable instructions that cause the hardware processor to receive an indication of an action being performed by a user on the system, cause the system to wait before updating the user interface in response to the action performed by the user, gather contextual information associated with the action while the system is waiting to process the action performed by the user to update the user interface, cause the system to update the user interface in response to the action performed by the user after gathering the contextual information, and store information indicative of the action and the contextual information.

Claims

1. A system, comprising:

at least one hardware processor;

a display coupled to the at least one hardware processor and configured to display a user interface; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform;

receiving an indication of an action being performed by a user on the system;

causing the system to wait before updating the user interface in response to the action performed by the user;

gathering contextual information associated with the action while the system is waiting to process the action performed by the user to update the user interface;

causing the system to update the user interface in response to the action performed by the user after gathering the contextual information; and

storing information indicative of the action and the contextual information.

2. The system of claim 1, wherein the action includes at least one of a keystroke and a click.

3. The system of claim 1, wherein gathering the contextual information includes gathering information regarding a state of the user interface when the user performed the action.

4. The system of claim 3, wherein gathering the information regarding the state of the user interface includes gathering information selected from the group consisting of:

a name of an active application, a title in a title bar of the active application, a value in at least one field of the active application, a size of a window of the active application, and a location of the window of the active application within the user interface.

5. The system of claim 1, further comprising a volatile memory and wherein storing the information includes storing the information indicative of the action and the contextual information in the volatile memory as an event.

6. The system of claim 5, further comprising a non-volatile memory and wherein the processor-executable instructions further cause the at least one hardware processor to perform:

writing one or more events stored in the volatile memory to an event log in the non-volatile memory.

7. The system of claim 6, wherein the processor-executable instructions further cause the at least one hardware processor to perform:

transmitting the event log to an external system.

8. The system of claim 5, wherein the processor-executable instructions further cause the at least one hardware processor to perform:

determining whether at least one event stored in the volatile memory includes personal information of the user.

9. The system of claim 8, wherein the personal information of the user includes information indicative of at least one member selected from the group consisting of:

a bank account of the user, a personal email account of the user, and a social media account of the user.

10. The system of claim 8, further comprising a non-volatile memory and wherein the processor-executable instructions further cause the at least one hardware processor to perform:

writing the at least one event stored in the volatile memory to an event log in the non-volatile memory responsive to a determination that the at least one event does not include personal information of the user.

11. The system of claim 1, wherein storing the information includes storing the information indicative of the action and the contextual information as an event in a plurality of events and wherein the processor-executable instructions further cause the at least one hardware processor to perform:

clustering the plurality of events to generate a plurality of clustered events;

identifying a plurality of sub-tasks in the plurality of clustered events, each of the plurality of sub-tasks comprising a sequence of one or more clustered events at least partially repeated in the plurality of clustered events;

identifying a task in the plurality of clustered events being performed by the user that comprises at least one sub-task of the plurality of sub-tasks; and

generating a score for the task indicative of a difficulty of automating the task.

12. The system of claim 11, wherein the processor-executable instructions further cause the at least one hardware processor to perform:

determining whether the score exceeds a threshold;

generating a first computer program that is configured to perform the task responsive to the score exceeding the threshold; and

executing the first computer program configured to perform the task responsive to generating the first computer program.

13. The system of claim 12, wherein the first computer program is configured to perform the task at least in part by controlling one or more other computer programs.

14. The system of claim 13, wherein the first computer program is configured to control the one or more other computer programs through at least one graphical user interface (GUI) of the one or more other computer programs.

15. The system of claim 13, wherein the task comprises a first sub-task and a second sub-task of the plurality of sub-tasks and wherein the first computer program is configured to control a second computer program to perform the first sub-task and control a third computer program to perform the second sub-task.

16. A method, performed by a computing device, for tracking tasks performed by a user of the computing device, the method comprising:

receiving an indication of an action being performed by the user on the computing device;

causing the computing device to wait before updating a user interface of the computing device responsive to the action performed by the user;

gathering contextual information associated with the action while the computing device is waiting to update the user interface responsive to the action performed by the user;

causing the computing device to update the user interface responsive to the action performed by the user; and

storing information indicative of the action and the contextual information.

17. The method of claim 16, wherein storing the information includes storing the information indicative of the action and the contextual information in a volatile memory of the computing device as an event.

18. The method of claim 17, further comprising:

determining whether at least one event stored in the volatile memory includes personal information of the user.

19. The method of claim 18, wherein determining whether the at least one event stored in the volatile memory includes personal information of the user includes determining whether the at least one event stored in the volatile memory comprises information indicative of at least one member selected from the group consisting of:

a bank account of the user, a personal email account of the user, and a social media account of the user.

20. The method of claim 18, further comprising:

writing the at least one event stored in the volatile memory to an event log in a non-volatile memory of the computing device responsive to a determination that the at least one event does not include personal information of the user.

21. The method of claim 20, further comprising:

transmitting the event log to an external system.

22. The method of claim 16, wherein storing the information includes storing the information indicative of the action and the contextual information as an event in a plurality of events and wherein the method further comprises:

clustering the plurality of events to generate a plurality of clustered events;

identifying a plurality of sub-tasks in the plurality of clustered events, each of the plurality of sub-tasks comprising a sequence of one or more clustered events at least partially repeated in the plurality of clustered events;

identifying a task in the plurality of clustered events being performed by the user that comprises at least one sub-task of the plurality of sub-tasks; and

generating a score for the task indicative of a difficulty of automating the task.

23. The method of claim 22, further comprising:

determining whether the score exceeds a threshold;

generating a computer program that is configured to perform the task responsive to the score exceeding the threshold; and

executing the computer program configured to perform the task responsive to generating the computer program.

24. The method of claim 23, wherein generating the computer program comprises generating a computer program that is configured to perform the task at least in part by controlling one or more other computer programs.

25. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform:

receiving an indication of an action being performed by the user on a computing device;

causing the computing device to wait before updating a user interface of the computing device responsive to the action performed by the user;

gathering contextual information associated with the action while the computing device is waiting to update the user interface responsive to the action performed by the user;

causing the computing device to update the user interface responsive to the action performed by the user; and

storing information indicative of the action and the contextual information.