AUTOMATED APPLICATION INTERACTION USING A VIRTUAL OPERATOR

• US 20100027896A1
• Filed: 09/09/2009
• Published: 02/04/2010
• Est. Priority Date: 06/28/2006
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Geva, Amir, Walach, Eugeniusz
• Application Number:
  US12/555,821
• Publication Number:
  US 20100027896A1
• Time in Patent Office: Days
• US Class Current: 382/229
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 21/36 by graphic or iconic representation
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

A computer-implemented method for automating interaction with a computer system includes linking a control computer system to an input interface and to an output interface of a client computer system, which is operative for producing user interface images on a display device. The control computer system executes distinct software modules that include a virtual operator for simulating actions of a human operator. Execution of the software modules causes the control computer system to capture an image from the output interface, and to recognize information in the image. In response to the information, the virtual operator controls an input device to automatically execute predetermined operations on the client computer system via the input interface.

Claims

1. A computer-implemented method for automating interaction with a computer system comprising:

linking a control computer system to an input interface of a client computer system and to an output interface of the client computer system operative for producing user interface images on a display device; and

executing with the control computer system distinct software modules that include a virtual operator for simulating actions of a human operator, wherein execution of the software modules causes the control computer system to perform the steps of;

capturing an image from the output interface;

recognizing information in the image; and

responsively to the information, controlling an input device by the virtual operator to thereby automatically execute predetermined operations on the client computer system via the input interface.

2. The method according to claim 1, wherein capturing the image from the output interface comprises receiving the image from an image scanner.

3. The method according to claim 1, wherein recognizing information in the image comprises verifying that the recognized information matches a pattern that is known to the virtual operator.

4. The method according to claim 1, wherein the predetermined operations comprise entering data.

5. The method according to claim 1, wherein recognizing information comprises optical character recognition.

6. The method according to claim 1, wherein the client computer system comprises a plurality of client computers, and the virtual operator comprises a plurality of instances of the virtual operator, the method comprising coordinating the plurality of instances of the virtual operator to concurrently control the plurality of client computers.

7. The method according to claim 1, wherein recognizing information in the image comprises providing a verification image comprising the recognized information to the human operator.

8. The method according to claim 1, wherein the predetermined operations comprise executing data input scripts comprising simulated device events.

9. A computer software product for automating interaction with a computer system, comprising a computer storage medium in which computer program instructions are stored, wherein the instructions comprise distinct modules that include a virtual operator for simulating actions of a human operator, which instructions, when executed by a control computer linked to an input interface of a client computer system and to an output interface of the client computer system operative for producing user interface images on a display device, cause the control computer to perform the steps of:

capturing an image from the output interface;

recognizing information in the image; and

responsively to the information, controlling an input device by the virtual operator to thereby automatically execute predetermined operations on the client computer system via the input interface.

10. The computer software product according to claim 9, wherein capturing the image from the output interface comprises receiving the image from an image scanner.

11. The computer software product according to claim 9, wherein recognizing information in the image comprises verifying that the recognized information matches a pattern that is known to the virtual operator.

12. The computer software product according to claim 9, wherein the predetermined operations comprise entering data.

13. The computer software product according to claim 9, wherein recognizing information comprises optical character recognition.

14. The computer software product according to claim 9, wherein the client computer system comprises a plurality of client computers, and the virtual operator comprises a plurality of instances of the virtual operator, the method comprising coordinating the plurality of instances of the virtual operator to concurrently control the plurality of client computers.

15. The computer software product according to claim 9, wherein recognizing information in the image comprises providing a verification image comprising the recognized information to the human operator.

16. The computer software product according to claim 9, wherein the predetermined operations comprise executing data input scripts comprising simulated device events.

17. A data processing system automating interaction with a computer system, comprising:

a client computer system operative for producing user interface images on a display device;

a control computer system linked to an input interface and to an output interface of the client computer system;

a processor; and

a memory accessible to the processor storing programs and data objects therein, wherein execution of the programs causes the processor to execute with the control computer system distinct software modules that include a virtual operator for simulating actions of a human operator, and to perform the steps of;

capturing an image from the output interface;

recognizing information in the image; and

responsively to the information, controlling an input device by the virtual operator to thereby automatically execute predetermined operations on the client computer system via the input interface.

18. The data processing system according to claim 17, wherein capturing the image from the output interface comprises receiving the image from an image scanner.

19. The data processing system according to claim 17, wherein recognizing information in the image comprises verifying that the recognized information matches a pattern that is known to the virtual operator.

20. The data processing system according to claim 17, wherein the predetermined operations comprise entering data.