System module of simulating machine operation screen based on non-invasive data-extraction system

• US 10,740,122 B2
• Filed: 07/20/2018
• Issued: 08/11/2020
• Est. Priority Date: 08/17/2017
• Status: Active Grant

• Current Assignee: ADLINK Technology Inc.
• Original Assignee: ADLINK Technology Inc.
• Inventors: Ng, Chua-Hong, Yang, Chao-Tung, Chen, Wei-Hung, Yu, Tsan-Ming, Lin, Shih-Hsun, Tseng, Yang-Chung, Hsu, Chih-Fu, Tu, Chien-Hsun, Wu, Ren-Yu, Lo, Chieh-Yuan, Shiao, Chih-Kai, Chang, Hsiao-Ling, Tseng, Te-Cheng, Chen, Chun-Liang
• Primary Examiner(s): Xia, Xuyang
• Application Number:
  US16/040,982
• Publication Number:
  US 20190056962A1
• Time in Patent Office: 753 Days
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A system module applied to the machine controller for simulating a machine operation screen based on a non-invasive data-extraction system, is disclosed. An image capture device of the system module can receive an original operation screen outputted from the machine controller, and transmit the original operation screen to the non-invasive data-extraction system and a high-speed image process unit for extraction of the information shown on the operation screen. The software control system can extract the operational information of the machine controller in real time, to create a machine operation flow for generating a simulated machine operation screen which is then outputted to a screen of the machine controller. As a result, the site working staff can be provided with operational information associated with the machine in real time, for example, the operational information includes currently executed operation screen, position of mouse cursor and pop-up window detection result.

Claims

1. A system module of simulating a machine operation screen based on an non-invasive data-extraction system, is applied to a machine controller and configured to control a machine, and the system module comprises:

an image capture device configured to receive an original operation screen outputted from a machine control program of the machine controller, and transmit the original operation screen to the non-invasive data-extraction system to extract information shown on the original operation screen of the machine;

a high-speed image process unit configured to receive the original operation screen of the machine, and perform an image process on the original operation screen to provide machine operational information, wherein the high-speed image process unit recognizes a specific page-pattern shown on the original operation screen and notes a page-ID, obtained from a recognition result, in an image of the original operation screen, and compares an icon of a mouse cursor on the original operation screen to position a coordinate of the mouse cursor, and define a threshold for an image change to detect occurrence of a pop-up window on the original operation screen;

a software control system configured to receive the information from the non-invasive data-extraction system, and analyze the machine operational information transmitted from the high-speed image process unit, wherein the software control system periodically updates an operational page of the original operation screen according to the page-ID, and compare a coordinate and a range of a control component embedded in the operational page, to create an operational flow of the machine controller for generating a simulated machine operation screen image, and wherein when the high-speed image process unit detects occurrence of the pop-up window, the high-speed image process unit outputs a warning message;

an image output device configured to output the simulated machine operation screen image linked with the control component, to a screen of the machine controller;

a signal receiving device configured to transmit a control signal of a keyboard/mouse inputted from the machine controller, to the software control system for processing, and

wherein the signal receiving device is configured to find the control component in the operational flow according to a position of mouse-clicking; and

an HID simulation device configured to simulate the keyboard/mouse to transmit a corresponding control signal to the machine controller, so as to enable the machine controller to execute an inputted command.

2. The system module according to claim 1, wherein the machine controller comprises a host computer and the screen electrically connected to the host computer, and the host computer is electrically connected to the keyboard/mouse.

3. The system module according to claim 1, wherein the machine controller comprises a machine operation program, and the machine operation program generates the original operation screen of the machine, and the operation screen comprises the operational pages of a main screen, a calibration screen, a program editing screen, a process execution monitoring screen, a warning screen, or a system parameter setting screen.

4. The system module according to claim 1, wherein the non-invasive data-extraction system is configured to receive the original operation screen image of the machine control program outputted from the machine controller, and perform region of interest extraction and optical character recognition on the original operation screen image, to convert information shown on the original operation screen image into digital information.

5. The system module according to claim 1, wherein the operation screen image captured by the image capture device of the system module is stored in a buffer built in the software control system to serve a temporarily-stored image which is to be outputted by the image output device to the machine controller.

6. The system module according to claim 1, wherein the high-speed image process unit is a post-stage processing unit for the image of the operation screen captured by the image capture device, and an image process algorithm is implemented in a FPGA, ASIC/ASSP, or a micro-processor.

7. The system module according to claim 1, wherein the specific page-pattern recognized by the high-speed image process unit of the system module is an image pattern preset by the user for each of the operational pages of the machine operation screen, and the high-speed image process unit transmits the image pattern to the software control system to analyze the site working staffs control operation for the machine through the machine controller, and the threshold for the image change is used to define whether the image of the operation screen becomes stable, thereby detecting the occurrence of the pop-up window.

8. The system module according to claim 1, wherein the software control system is operated in one of an editing mode and an execution mode, and in the editing mode, the software control system creates the operational flow of the machine controller, and the operational flow is configured to describe relationships between the operational pages of the machine operation screen;

wherein in the execution mode, the software control system is based on the signal receiving device and the operation flow to analyze the information of operation performed on the machine by the site working staff, so as to generate the simulated machine operation screen image, and the HID simulation device transmits the control signal generated in response to the control component, to the machine controller.

9. The system module according to claim 8, wherein, in the editing mode, the software control system plans information of each of the operational pages, and the planned information comprises:

the page-ID shown on the operation screen for representing the operational page;

the image of the operation screen used to compare the control component embedded in the operation screen for creating the operational flow of the machine controller;

a coordinate and a range of the control component for labelling the control component used to switch to other the operational page; and

a coordinate and a range of a home/return control component for labelling the control component used to switch to a home page or return page.

10. The system module according claim 9, wherein the software control system automatically labels the control component on the operation screen of the machine, and find the home/return control component according to predefined image patterns, and a HID simulation device transmits the command corresponding to the control component, to the machine controller, and the operational pages of the operation screen is browsed one by one according to Tree Traversal Algorithm, thereby creating the operational flow chart of the machine operation program, and the operational flow chart records the operational pages, the control components embedded in the operational pages, and a target operational page in response to an action of clicking the control component.

11. The system module according to claim 1, wherein the software control system stores the image of the operation screen captured by the image capture device, and analyzes the updated state of the machine operational information transmitted from the high-speed image process unit, and the updated state comprises a position of the mouse cursor, the operational page of currently-executed the operation screen of the machine control program, and a record of the control components of the operational pages corresponding to the operational flow.

12. The system module according to claim 1, wherein when the high-speed image process unit detects occurrence of the pop-up window, the software control system reports the warning message comprising error message-box or warn message-box, to a central monitoring system, and after the HID simulation device releases the warning message, the non-invasive data-extraction system continuously performs the process of extracting the data shown the operation screen.

13. The system module according to claim 1, wherein the software control system receives the control signal transmitted from the signal receiving device, and compares the range of the control component of the machine operation flow and the position of mouse clicking in response of the site working staffs operation of mouse clicking, and when the software control system confirms the position of mouse clicking within an effective range of the control component according to the machine operation flow, the image output device outputs the simulated machine operation screen linked with the clicked control component, to the screen of the machine controller.

14. The system module according to claim 1, wherein the software control system temporarily stores the data, which is inputted by the site working staff through keyboard, in the signal receiving device;

and

wherein after the operational page is switched completely, the HID simulation device performs actual data input to the machine controller.