Robotic process automation

• US 9,555,544 B2
• Filed: 04/08/2016
• Issued: 01/31/2017
• Est. Priority Date: 07/02/2015
• Status: Active Grant

• Current Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Original Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Inventors: Bataller, Cyrille, Jacquot, Adrien, Torres, Sergio Ral
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US15/094,063
• Publication Number:
  US 20170001308A1
• Time in Patent Office: 298 Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1679 characterised by the tasks executed
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/0423 Input/output
• CPC Class Codes: G05B 2219/36184 Record actions of human expert, teach by showing
• CPC Class Codes: G05B 2219/40116 Learn by operator observation, symbiosis, show, watch
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automating a manual process. The methods, systems, and apparatus include actions of identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with another computer. Additional actions include obtaining one or more images taken of a display of the computer while the user is interacting with the computer in manually performing the process and applying a computer vision technique to identify one or more activities associated with the process. Further actions include, for each of the one or more identified activities, generating activity information associated with the activity and generating a process definition for use in causing the robot to automatically perform the process.

Claims

1. A computer-implemented method comprising:

identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer;

obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process;

applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change;

in response to determining that the change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies the predetermined threshold corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process;

for each of the identified activities, generating activity information associated with the activity;

generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and

storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer.

2. The method of claim 1, wherein obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process comprises:

obtaining images taken of the display of the computer from a camera.

3. The method of claim 1, wherein the one or more images comprise a video.

4. The method of claim 1, wherein generating activity information associated with the activity comprises:

generating a snapshot of a portion of a screen shown on the display before the activity is performed.

5. The method of claim 1, wherein generating activity information associated with the activity comprises:

generating a screenshot of a screen shown on the display after the activity is performed.

6. The method of claim 1, wherein the activities comprise two or more of a key press, a mouse click, a screen touch, a process in a foreground change, or a scene change.

7. The method of claim 1, comprising:

accessing the process definition; and

automatically instructing the robot to physically interact with the computer or the other computer based on the activities and activity information indicated by the process definition.

8. The method of claim 7, wherein automatically instructing the robot to interact with the computer or the other computer based on the activities and activity information indicated by the process definition comprises:

identifying a portion of a screen shown on another display of the other computer that visually matches a snapshot indicated by the activity information for a particular activity; and

instructing the robot to physically touch the other display at a location corresponding to the center of the portion.

9. The method of claim 7, wherein automatically instructing the robot to interact with the other computer based on the activities and activity information indicated by the process definition comprises:

determining that a screen shown on the other display corresponds to a screenshot indicated by the activity information for a particular activity; and

in response to determining that the screen shown on the other display corresponds to a screenshot indicated by the activity information for a particular activity, instructing the robot to physically interact with the other computer based on a subsequent activity indicated by the process definition.

10. The method of claim 7, wherein automatically instructing the robot to interact with the computer or the other computer based on the activities and activity information indicated by the process definition comprises:

identifying a portion of a screen shown on another display of the other computer that visually matches a snapshot indicated by the activity information for a particular activity; and

instructing the robot to provide an electronic signal to the other computer to receive a click on coordinates corresponding to the center of the portion of the screen that visually matches the snapshot.

11. The method of claim 1, comprising:

generating a graphical version of the process definition that a person can use to modify the process.

12. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer;

obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process;

applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change;

in response to determining that the change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies the predetermined threshold corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process;

for each of the identified activities, generating activity information associated with the activity;

generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and

storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer.

13. The system of claim 12, wherein obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process comprises:

obtaining images taken of the display of the computer from a camera.

14. The system of claim 12, wherein the one or more images comprise a video.

15. The system of claim 12, wherein generating activity information associated with the activity comprises:

generating a snapshot of a portion of a screen shown on the display before the activity is performed.

16. The system of claim 12, wherein generating activity information associated with the activity comprises:

generating a screenshot of a screen shown on the display after the activity is performed.

17. The system of claim 12, wherein the activities comprise tow or more of a key press, a mouse click, a screen touch, a process in a foreground change, or a scene change.

18. The system of claim 12, comprising:

accessing the process definition; and

automatically instructing the robot to physically interact with the computer or the other computer based on the activities and activity information indicated by the process definition.

19. The system of claim 18, wherein automatically instructing the robot to interact with the computer or the other computer based on the activities and activity information indicated by the process definition comprises:

identifying a portion of a screen shown on another display of the other computer that visually matches a snapshot indicated by the activity information for a particular activity; and

instructing the robot to physically touch the other display at a location corresponding to the center of the portion.

20. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer;

obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process;

applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change;

in response to determining that the change in images taken of the display of the computer, while the user is interacting with the computer in manually performing the process satisfies the redetermined threshold cones corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process;

for each of the identified activities, generating activity information associated with the activity;

generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and

storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer.