Detection and definition of virtual objects in remote screens

• US 10,769,427 B1
• Filed: 04/19/2018
• Issued: 09/08/2020
• Est. Priority Date: 04/19/2018
• Status: Active Grant

• Current Assignee: Automation Anywhere, Inc.
• Original Assignee: Automation Anywhere, Inc.
• Inventors: Gajera, Prakash, Patel, Gaurang, Kakhandiki, Abhijit
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US15/957,030
• Time in Patent Office: 873 Days
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/164 Noise filtering
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

Methods and systems that detect and define virtual objects in remote screens which do not expose objects. This permits simple and reliable automation of existing applications. In certain aspects a method for detecting objects from an application program that are displayed on a computer screen is disclosed. An image displayed on the computer screen is captured. The image is analyzed to identify blobs in the image. The identified blobs are filtered to identify a set of actionable objects within the image. Optical character recognition is performed on the image to detect text fields in the image. Each actionable object is linked to a text field positioned closest to a left or top side of the actionable object. The system automatically detects the virtual objects and links each actionable object such as textboxes, buttons, checkboxes, etc. to the nearest label object.

Claims

1. A computer-implemented method for automating usage of an application program comprising:

detecting objects and text from a first screen of information generated by the application program for display on a computer monitor by,

capturing a first image of the screen of information;

generating a first signature to identify the first image;

analyzing the first image to identify a set of actionable objects within the first image;

performing optical character recognition to detect text fields in the first image;

linking each identified actionable object to each detected text field positioned proximate to the identified actionable object in the first image;

in a screen signature database, storing for the first image, the first signature, which is associated with a smart screen, wherein the smart screen stores the links between each actionable object and its correspondingly linked text field; and

subsequently interacting with the application, and for a second screen of information displayed by the application,

capturing a second image of a second screen of information displayed by the application program,

generating a second signature to identify the second image;

in the screen signature database, searching for a previously stored signature, which comprises at least the first signature, that matches the second signature,

if the second signature matches a previously stored signature, then using the stored matching signature to retrieve and utilize a previously stored smart screen, that is associated with the stored matching signature, to process the second image.

2. The computer-implemented method of claim 1 wherein the operation of linking each identified actionable object to each detected text field positioned proximate to the identified actionable object comprises:

retrieving common patterns and applying the common labels to link each identified actionable object to each detected text field positioned proximate to the identified actionable object.

3. The computer-implemented method of claim 2 further comprising:

utilizing a machine learning engine to detect the common patterns and further training the machine learning engine based on the detected common patterns to cause detection of objects in the second image with corresponding objects in a previously stored smart screen where the position of the objects differs between the second image and the previously stored smart screen.

4. The computer-implemented method of claim 1 further comprising:

if the second signature does not match the previously stored signature, then

analyzing the second image to identify a set of actionable objects within the second image;

performing optical character recognition to detect text fields in the second image;

linking each identified actionable object to each detected text field positioned proximate to the identified actionable object in the second image; and

storing in the screen signature database, for the second image, the new signature, which is associated with a second smart screen, wherein the second smart screen stores the links between each actionable object and its correspondingly linked text field.

5. The computer-implemented method of claim 1 further comprising storing for the first image to the screen signature database:

an associated object position identifier identifying a position of each actionable object on the first image; and

a text field position identifier identifying a position of each text field on the first image.

6. The computer-implemented method of claim 1 further comprising removing background noise from the first image before analyzing the first image to identify a set of actionable objects within the first image.

7. The computer-implemented method of claim 1 wherein the application program generates multiple screens of information, the method further comprising, performing the operations of capturing, generating, analyzing, performing optical character recognition and linking for each screen of information generated by the application program.

8. The computer-implemented method of claim 1 wherein analyzing the first image to identify a set of actionable objects within the first image comprises:

analyzing the first image to identify blobs in the first image;

filtering the identified blobs to identify the set of actionable objects by retrieving one or more predefined filtering criteria that cause blobs larger or smaller than predefined sizes to be filtered out.

9. A robotic process automation system comprising:

data storage that stores a screen signature database; and

a processor operatively coupled to the data storage and configured to perform operations that when executed cause the processor to automate usage of an application program by detecting objects and text from a first screen of information generated by the application program for display on a computer monitor by;

capturing a first image of the screen of information;

generating a first signature to identify the first image;

analyzing the first image to identify a set of actionable objects within the first image;

performing optical character recognition to detect text fields in the first image;

linking each identified actionable object to each detected text field positioned proximate to the identified actionable object in the first image;

in the screen signature database, storing for the first image, the first signature, which is associated witha smart screen, wherein the smart screen stores the links between each actionable object and its correspondingly linked text field; and

subsequently interacting with the application, and for a second screen of information displayed by the application,

capturing a second image of a second screen of information displayed by the application program,

generating a second signature to identify the second image;

in the screen signature database, searching for a previously stored signature, which comprises at least the first signature, that matches the second signature,

if the second signature matches a previously stored signature, then using the stored matching signature to retrieve and utilize a previously stored smart screen, that is associated with the stored matching signature, to process the second image.

10. The robotic process automation system of claim 9 wherein the operation of linking each identified actionable object to each detected text field positioned proximate to the identified actionable object comprises:

retrieving common patterns and applying the common labels to link each identified actionable object to each detected text field positioned proximate to the identified actionable object.

11. The robotic process automation system of claim 10 further comprising:

utilizing a machine learning engine to detect the common patterns and further training the machine learning engine based on the detected common patterns to cause detection of objects in the second image with corresponding objects in a previously stored smart screen where the position of the objects differs between the second image and the previously stored smart screen.

12. The robotic process automation system of claim 9 wherein the processor is further programmed to perform the operation of:

if the second signature does not match the previously stored signature, then

analyzing the second image to identify a set of actionable objects within the second image;

performing optical character recognition to detect text fields in the second image;

linking each identified actionable object to each detected text field positioned proximate to the identified actionable object in the second image; and

storing in the screen signature database, for the second image, the new signature, which is associated with a second smart screen, wherein the second smart screen stores the links between each actionable object and its correspondingly linked text field.

13. The robotic process automation system of claim 9 wherein the processor is further programmed to perform the operation of storing for the first image to the screen signature database:

an associated object position identifier identifying a position of each actionable object on the first image; and

a text field position identifier identifying a position of each text field on the first image.

14. The robotic process automation system of claim 9 wherein the processor is further programmed to perform the operation of removing background noise from the first image before analyzing the first image to identify a set of actionable objects within the first image.

15. The robotic process automation system of claim 9 wherein the application program generates multiple screens of information, and wherein the processor is further programmed to perform the operation of performing the operations of capturing, generating, analyzing, performing optical character recognition and linking for each screen of information generated by the application program.

16. The robotic process automation system of claim 9 wherein the processor is further programmed to perform the operation of analyzing the first image to identify a set of actionable objects within the first image by:

analyzing the first image to identify blobs in the first image;

filtering the identified blobs to identify the set of actionable objects by retrieving one or more predefined filtering criteria that cause blobs larger or smaller than predefined sizes to be filtered out.

17. A tangible storage medium, having stored thereupon one or more program modules comprising computer-executable instructions for execution on a computer system, the computer-executable instructions executing on a server processor to cause the computer system to implement a computer-implemented method for automating usage of an application program comprising:

detecting objects and text from a first screen of information generated by the application program for display on a computer monitor by,

capturing a first image of the screen of information;

generating a first signature to identify the first image;

analyzing the first image to identify a set of actionable objects within the first image;

performing optical character recognition to detect text fields in the first image;

linking each identified actionable object to each detected text field positioned proximate to the identified actionable object in the first image;

in a screen signature database, storing for the first image, the first signature, which is associated with a smart screen, wherein the smart screen stores the links between each actionable object and its correspondingly linked text field; and

subsequently interacting with the application, and for a second screen of information displayed by the application,

capturing a second image of a second screen of information displayed by the application program,

generating a second signature to identify the second image;

in the screen signature database, searching for a previously stored signature, which comprises at least the first signature, that matches the second signature, if the second signature matches a previously stored signature, then using the stored matching signature to retrieve and utilize a previously stored smart screen, that is associated with the stored matching signature, to process the second image.

18. The tangible storage medium of claim 17 wherein the operation of linking each identified actionable object to each detected text field positioned proximate to the identified actionable object comprises:

retrieving common patterns and applying the common labels to link each identified actionable object to each detected text field positioned proximate to the identified actionable object.

19. The tangible storage medium of claim 18 wherein the computer-implemented method for automating usage of an application program further comprises:

utilizing a machine learning engine to detect the common patterns and further training the machine learning engine based on the detected common patterns to cause detection of objects in the second image with corresponding objects in a previously stored smart screen where the position of the objects differs between the second image and the previously stored smart screen.

20. The tangible storage medium of claim 17 wherein the computer-implemented method for automating usage of an application program further comprises, if the second signature does not match the previously stored signature, then:

analyzing the second image to identify a set of actionable objects within the second image;

performing optical character recognition to detect text fields in the second image;

linking each identified actionable object to each detected text field positioned proximate to the identified actionable object in the second image; and

storing in the screen signature database, for the second image, the new signature, which is associated with a second smart screen, wherein the second smart screen stores the links between each actionable object and its correspondingly linked text field.