Robotic process automation
First Claim
1. A computer-implemented method comprising:
- identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer;
obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process;
applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change;
in response to determining that the change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies the predetermined threshold corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process;
for each of the identified activities, generating activity information associated with the activity;
generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and
storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer.
1 Assignment
0 Petitions
Accused Products
Abstract
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automating a manual process. The methods, systems, and apparatus include actions of identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with another computer. Additional actions include obtaining one or more images taken of a display of the computer while the user is interacting with the computer in manually performing the process and applying a computer vision technique to identify one or more activities associated with the process. Further actions include, for each of the one or more identified activities, generating activity information associated with the activity and generating a process definition for use in causing the robot to automatically perform the process.
-
Citations
20 Claims
-
1. A computer-implemented method comprising:
-
identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer; obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process; applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change; in response to determining that the change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies the predetermined threshold corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process; for each of the identified activities, generating activity information associated with the activity; generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. A system comprising:
-
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising; identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer; obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process; applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change; in response to determining that the change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies the predetermined threshold corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process; for each of the identified activities, generating activity information associated with the activity; generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer. - View Dependent Claims (13, 14, 15, 16, 17, 18, 19)
-
-
20. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
-
identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer; obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process; applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change; in response to determining that the change in images taken of the display of the computer, while the user is interacting with the computer in manually performing the process satisfies the redetermined threshold cones corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process; for each of the identified activities, generating activity information associated with the activity; generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer.
-
Specification