Method for monitoring a graphical user interface on a second computer display from a first computer

• US 7,870,504 B1
• Filed: 10/01/2003
• Issued: 01/11/2011
• Est. Priority Date: 10/01/2003
• Status: Active Grant

• Current Assignee: Eggplant Software Incorporated (Keysight Technologies, Inc.)
• Original Assignee: Testplant, Inc. (Keysight Technologies, Inc.)
• Inventors: McIntosh, John W., Simons, Douglas P., Bieber, Ray L., Gillaspie, Jonathan D.
• Primary Examiner(s): Pesin; Boris
• Assistant Examiner(s): HEFFINGTON, JOHN M
• Application Number:
  US10/678,043
• Time in Patent Office: 2,659 Days
• Field of Search: 715/740, 715/704
• US Class Current: 715/803
• CPC Class Codes: G03G 15/50 Machine control of apparatus for electrographic processes using a charge pattern, e.g. regulating differents parts of the machine, multimode copiers, microprocessor control sequencing control G03G21/14
• CPC Class Codes: G06F 11/2273 Test methods
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/1415 with means for detecting differences between the image stored in the host and the images displayed on the displays
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 9/004 Predictors, e.g. intraframe, interframe coding

Abstract

A high level interface between a remote computer and local computer operator permits the remote computer to be controlled via a sequence of interactions. The remote computer may be monitored for display information which is expected, and also controlled in the event the expected information either is or is not obtained. Command language extensions are provided which extend and complement a basic scripting language. Scripts with embedded command language extensions may be executed through a remote interface, permitting remote testing, operation and evaluation. The development of the scripts, including embedded command language extensions, may be prepared through execution of commands at the local computer on a depiction of the GUI received from the remote computer, and subsequently saved as a script.

Claims

1. A method for using a first computer system to remotely monitor and interact with the operation of a second computer system through a graphical user interface of said second computer system, comprising the steps of:

receiving a bitmap image of said second computer system graphical user interface at said first computer system;

searching said bitmap image of said second computer system graphical user interface for a first graphical element contained within and comprising less than said bitmap image through an automated execution of said first computer system commands;

responsive to said receiving step and results of said searching step, generating a user peripheral input device input action within said second computer system graphical user interface as interpreted by said second computer by automatically creating and passing a signal through a communications channel from said first computer system to said second computer system graphical user interface;

monitoring said bitmap image of said second computer system graphical user interface automatically from said first computer system for an expected second graphical element contained within and comprising less than said bitmap image within a predetermined time interval; and

signaling a failure at said first computer system if said predetermined time interval elapses without detecting said expected second graphical element.

2. The method of claim 1 further comprising the steps of:

providing graphical user interface language extensions commands to a scripting language; and

passing said generated user input action through said graphical user interface language extensions from a scripting language processor to a language extensions processor.

3. The method of claim 1 further comprising the steps of:

generating a user input action within said second computer system responsive to said second graphical element;

monitoring and searching said second computer system graphical user interface automatically from said first computer system for an expected third graphical element contained within and comprising less than said bitmap image within a predetermined time interval; and

signaling a failure at said first computer system if said predetermined time interval elapses without detecting said expected third graphical element.

4. The method of claim 1 further comprising the steps of:

depicting said second computer system graphical user interface upon a local display of said first computer system including said first graphical element; and

receiving a local user input action at said first computer system within said local display;

wherein said generated user input action emulates said local user input action.

5. The method of claim 1 further comprising the steps of:

providing graphical user interface language extensions commands to a scripting language;

depicting said computer system graphical user interface upon a local display of said first computer system including said first graphical element;

receiving a local user input action within said local display;

transferring said user input action to a script stored on said first computer system;

passing said generated user input action through said graphical user interface language extensions from a scripting language processor to a language extensions processor for reproduction at said second computer system graphical user interface, wherein said generated user input action emulates said local user input action; and

re-executing said steps of receiving, searching, generating, monitoring and signaling subsequent to said storing step under control of said stored script.

6. The method of claim 1, wherein said step of generating a user peripheral input device input action further comprises locating said user peripheral input device input action within said first graphical element.

7. The method of claim 6, wherein said user peripheral input device input action further comprises a click event.

8. The method of claim 1, further comprising the step of converting said received bitmap image of said second computer system graphical user interface into a variant different from said second computer system graphical user interface prior to said searching step.

9. The method of claim 8, wherein said step of converting further comprises color translations.

10. The method of claim 9, wherein said color translations further comprises mapping and conversion of said second computer system graphical user interface into particular color sets based upon geometry.

11. The method of claim 1, wherein said step of searching further comprises determining the location of said first graphical element when said first graphical element is found within said second computer system graphical user interface.

12. The method of claim 1, wherein said step of searching further comprises searching for additional occurrences of said first graphical element within said second computer system graphical user interface when said first graphical element is found within said second computer system graphical user interface.

13. A method for enabling a local system to automatically remotely operate a remote computer system through a graphical user interface on said remote computer system by using local scripts that selectively respond to changes in graphical displays upon said graphical user interface of said remote computer system, comprising the steps of:

displaying a depiction of said remote system graphical user interface display on said local system;

capturing user input effected in said depiction of said remote system graphical user interface display;

implementing automatically through a local system command language set user input emulations that are representative of said captured user input when reproduced at said remote computer system graphical user interface through a peripheral input device channel;

image processing said remote computer system graphical displays automatically using a local system script that searches for and detects a first entity contained within and comprising less than said graphical display upon said graphical user interface of said remote computer system;

controlling a flow of execution of said local system automatically through a scripting language having scripting commands in combination with said command language set, said flow which varies responsive to a result of detection of said first entity during said image processing step; and

communicating said user input emulations between said local system and said remote computer system graphical user interface through a communication interface responsive to said flow controlling step.

14. The method for enabling a local system to remotely operate a remote computer system through a graphical user interface on said remote computer system of claim 13 further comprising the steps of:

storing said scripting commands into a storing means;

inserting a command from said command language set into said storing means; and

executing said inserted stored command.

15. The method of claim 13, wherein said step of communicating between said local system and said remote computer system graphical user interface further comprises locating a user peripheral input device input action at a location relative to said first entity.

16. The method of claim 15, wherein said user peripheral input device input action further comprises a click event.

17. The method of claim 13, further comprising the step of converting said depiction of said remote system graphical user interface into a variant different from said remote system graphical user interface prior to said searching step.

18. The method of claim 17, wherein said step of converting further comprises color translations.

19. The method of claim 18, wherein said color translations further comprises mapping and conversion of said remote system graphical user interface into particular color sets based upon geometry.

20. The method of claim 13, wherein said step of image processing further comprises determining the location of said first graphical entity when said first graphical entity is found within said remote system graphical user interface.

21. The method of claim 13, wherein said step of image processing further comprises detecting additional occurrences of said first graphical entity within said remote system graphical user interface when said first graphical entity is found within said remote system graphical user interface.

22. A method for using a first computer system to remotely monitor and interact with the operation of a second computer system through a graphical user interface of said second computer system, comprising the steps of:

receiving a representation of said second computer system graphical user interface at said first computer system;

searching said representation of said second computer system graphical user interface for a first graphical entity contained within and comprising less than said representation of said second computer system graphical user interface through an automated execution of said first computer system commands; and

responsive to said receiving step and results of said searching step, generating a user peripheral input device input action within said second computer system graphical user interface as interpreted by a second computer by automatically creating and passing a signal through a communications channel from said first computer system to said second computer system graphical user interface.

23. The method of claim 22 further comprising the steps of:

providing graphical user interface language extensions commands to a scripting language; and

passing said generated user input action through said graphical user interface language extensions from a scripting language processor to a language extensions processor.

24. The method of claim 22 further comprising the steps of:

depicting said second computer system graphical user interface upon a local display of said first computer system including said first graphical entity; and

receiving a local user input action at said first computer system within said local display;

wherein said generated user input action emulates said local user input action.

25. The method of claim 22, wherein said step of generating a user peripheral input device input action further comprises locating said user peripheral input device input action within said first graphical entity.

26. The method of claim 25, wherein said user peripheral input device input action further comprises a click event.

27. The method of claim 22, further comprising the steps of:

monitoring and searching said second computer system graphical user interface automatically from said first computer system for an expected second graphical entity within a predetermined time interval; and

signaling a failure at said first computer system if said predetermined time interval elapses without detecting said expected second graphical entity.

28. The method of claim 27 further comprising the steps of:

generating a user input action within said second computer system responsive to said second graphical entity;

monitoring and searching said second computer system graphical user interface for an expected third graphical entity within a predetermined time interval; and

signaling a failure at said first computer system if said predetermined time interval elapses without detecting said expected third graphical entity.

29. The method of claim 27 further comprising the steps of:

providing graphical user interface language extensions commands to a scripting language; and

depicting said computer system graphical user interface upon a local display of said first computer system including said first graphical entity;

receiving a local user input action within said local display;

transferring said user input action as a command to a script stored on said first computer system;

passing said generated user input action command through said graphical user interface language extensions from said a scripting language processor to a language extensions processor for reproduction at said second computer system graphical user interface, wherein said generated user input action command emulates said local user input action; and

re-executing said steps of receiving, generating, monitoring and signaling subsequent to said storing step under control of said stored script.

30. The method of claim 22, further comprising the step of converting said received representation of said second computer system graphical user interface into a variant different from said second computer system graphical user interface prior to said searching step.

31. The method of claim 30, wherein said step of converting further comprises color translations.

32. The method of claim 31, wherein said color translations further comprises mapping and conversion of said second computer system graphical user interface into particular color sets based upon geometry.

33. The method of claim 22, wherein said step of searching further comprises determining the location of said first graphical entity when said first graphical entity is found within said second computer system graphical user interface.

34. The method of claim 22, wherein said step of searching further comprises searching for additional occurrences of said first graphical entity within said second computer system graphical user interface when said first graphical entity is found within said second computer system graphical user interface.

35. A method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine by using computing scripts that selectively respond to changes within graphical displays upon said graphical user interface of said second computing machine, comprising the steps of:

displaying a depiction of said second computing machine graphical user interface display on a graphical user interface of said first computing machine;

capturing user input effected in said depiction of said second computing machine graphical user interface display;

image processing said second computing machine graphical displays using a first computing machine script that searches for and detects the presence of a first entity contained within and comprising less than said graphical display upon said graphical user interface of said second computing machine;

controlling a flow of execution of said first computing machine through a scripting language having scripting commands in combination with said a command language set, said flow which varies responsive to a result of detection of said first entity during said image processing step; and

implementing user input commands at said second computing machine graphical user interface emulating said captured user input by communicating between said first computing machine and said second computing machine graphical user interface through a communication interface responsive to said flow controlling step, a first user input command implemented responsive to a detection of said first entity during said image processing step and a second user input command different from said first user input command implemented responsive to a non-detection of said first entity during said image processing step.

36. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 35 wherein said step of implementing user input commands further comprises locating user input commands at a location determined relative to said first entity.

37. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 36 wherein said step of implementing user input commands further comprises locating user input commands directly upon said first entity.

38. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 35 wherein said first entity further comprises an icon.

39. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 35 wherein said first entity further comprises a graphical control.

40. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 35 wherein said first entity further comprises a prompt.

41. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 35 wherein said first entity further comprises a command button.

42. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 35 wherein said first entity further comprises a message box.

43. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 35 wherein said second computing machine further comprises a virtual network computing component.

44. The method of claim 35, further comprising the step of converting said depiction of said second computing machine graphical user interface into a variant different from said second computing machine graphical user interface prior to said searching.

45. The method of claim 44, wherein said step of converting further comprises color translations.

46. The method of claim 45, wherein said color translations further comprises mapping and conversion of said second computing machine graphical user interface into particular color sets based upon geometry.

47. The method of claim 35, wherein said searching further comprises determining the location of said first entity when said first entity is found within said second computing machine graphical user interface.

48. The method of claim 35, wherein said searching further comprises searching for additional occurrences of said first entity within said second computing machine graphical user interface when said first entity is found within said second computing machine graphical user interface.

49. A method for using a first computer system to remotely monitor and interact with the operation of a second computer system through a graphical user interface of said second computer system, comprising the steps of:

receiving a bitmap image of said second computer system graphical user interface at said first computer system;

searching said bitmap image of said second computer system graphical user interface for a first graphical element contained within and comprising less than said bitmap image through an automated execution of said first computer system commands;

responsive to said receiving step and a not-found result of said searching step, generating a first user peripheral input device input action within said second computer system graphical user interface by automatically creating and passing a signal through a communications channel from said first computer system to said second computer system graphical user interface;

and responsive to said receiving step and a found result of said searching step, generating a second user peripheral input device input action within said second computer system graphical user interface different from said first user peripheral input device input action by automatically creating and passing a signal through a communications channel from said first computer system to said second computer system graphical user interface.

50. The method for using a first computer system to remotely monitor and interact with the operation of a second computer system through a graphical user interface of said second computer system of claim 49, further comprising the steps of:

monitoring and searching said bitmap image of said second computer system graphical user interface automatically from said first computer system for an expected second graphical element contained within and comprising less than said image within a predetermined time interval; and

signaling a failure at said first computer system if said predetermined time interval elapses without detecting said expected second graphical element.

51. The method of claim 49, further comprising the step of converting said received representation of said second computer system graphical user interface into a variant different from said second computer system graphical user interface prior to said searching step.

52. The method of claim 51, wherein said step of converting further comprises color translations.

53. The method of claim 52, wherein said color translations further comprises mapping and conversion of said second computer system graphical user interface into particular color sets based upon geometry.

54. The method of claim 49, wherein said step of searching further comprises determining the location of said first graphical element when said first graphical element is found within said second computer system graphical user interface.

55. The method of claim 49, wherein said step of searching further comprises searching for additional occurrences of said first graphical element within said second computer system graphical user interface when said first graphical element is found within said second computer system graphical user interface.

56. A method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine by using computing scripts that selectively respond to changes within graphical displays upon said graphical user interface of said second computing machine, comprising the steps of:

displaying a depiction of said second computing machine graphical user interface display on a graphical user interface of said first computing machine;

image processing said second computing machine graphical displays using a first computing machine search command set that searches for and detects the presence of a first entity contained within and comprising less than said graphical display upon said graphical user interface of said second computing machine;

controlling a flow of execution of said first computing machine through a scripting language having scripting commands in combination with a command language set, said flow which varies responsive to a result of detection of said first entity during said image processing step; and

communicating between said first computing machine and said second computing machine graphical user interface through a communication interface responsive to said flow controlling step, a first user input command implemented at said second computing machine responsive to a detection of said first entity during said image processing step and a second user input command different from said first user input command implemented at said second computing machine responsive to a non-detection of said first entity during said image processing step.

57. The method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine of claim 56, wherein said first and second user input commands are implemented at said second computing machine graphical user interface by transmitting signals from said first computing machine to said second computing machine graphical user interface through a peripheral input device channel.

58. The method of claim 56, further comprising the step of converting said depiction of said second computing machine graphical user interface into a variant different from said second computing machine graphical user interface prior to said searching.

59. The method of claim 58, wherein said step of converting further comprises color translations.

60. The method of claim 59, wherein said color translations further comprises mapping and conversion of said second computing machine graphical user interface into particular color sets based upon geometry.

61. The method of claim 56, wherein said searching further comprises determining the location of said first entity when said first entity is found within said second computing machine graphical user interface.

62. The method of claim 56, wherein said searching further comprises searching for additional occurrences of said first entity within said second computing machine graphical user interface when said first entity is found within said second computing machine graphical user interface.

63. A method for enabling a first computing machine to remotely operate a second computing machine through a graphical user interface on said second computing machine by using computing scripts that selectively respond to changes within graphical displays upon said graphical user interface of said second computing machine, comprising the steps of:

displaying a depiction of said second computing machine graphical user interface display on a graphical user interface of said first computing machine;

capturing user input effected in said depiction of said second computing machine graphical user interface display;

developing at least one computing script incorporating user input into said first computing machine selected from ones of various scripting functions and commands, said at least one computing script further incorporating said captured user input;

image processing said second computing machine graphical displays using a first computing machine search command set that searches for and detects the presence of a first entity contained within and comprising less than said graphical display upon said graphical user interface of said second computing machine;

controlling a flow of execution of said first computing machine through said at least one computing script, said flow which varies responsive to a result of detection of said first entity during said image processing step; and

implementing user input commands at said second computing machine graphical user interface emulating said captured user input by communicating between said first computing machine and said second computing machine graphical user interface through a communication interface responsive to said flow controlling step, a first user input command implemented at said second computing machine graphical user interface responsive to a detection of said first entity during said image processing step and a second user input command different from said first user input command implemented at said second computing machine graphical user interface responsive to a non-detection of said first entity during said image processing step.

64. The method of claim 63, further comprising the step of converting said depiction of said second computing machine graphical user interface into a variant different from said second computing machine graphical user interface prior to said searching step.

65. The method of claim 64, wherein said step of converting further comprises color translations.

66. The method of claim 65, wherein said color translations further comprises mapping and conversion of said second computing machine graphical user interface into particular color sets based upon geometry.

67. The method of claim 63, wherein said searching further comprises determining the location of said first entity when said first entity is found within said second computing machine graphical user interface.

68. The method of claim 63, wherein said step of searching further comprises searching for additional occurrences of said first entity within said second computing machine graphical user interface when said first entity is found within said second computing machine graphical user interface.