GENERATING SUMMARY OF ACTIVITY ON COMPUTER GUI

• US 20160349928A1
• Filed: 05/27/2015
• Published: 12/01/2016
• Est. Priority Date: 05/27/2015
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Li, Linlin, Soenaryo, Edwin B.
• Application Number:
  US14/722,215
• Publication Number:
  US 20160349928A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

Auditing activity of a user on a user computing device is provided. When the user computing device receives an input in a graphical user interface (GUI) from a human interface device (HID), an active window is identified within a captured screenshot by locating a screen location of the HID input in the GUI. The active window along with an extracted textual title of the active window, the screen location of the HID input and a user identifier are stored to a data store.

Claims

1. A method for auditing activity of a user on a user computing device, the method comprising:

receiving, by a computer, a screenshot of a display device of the user computing device when the user computing device receives an input in a graphical user interface (GUI) from a human interface device (HID), and a screen location of the HID input, and a user identifier;

identifying, by the computer, an active window in the received screenshot, based on the screen location of the HID input;

extracting, by the computer, from the screenshot, the identified active window;

determining, by the computer, a textual title of the active window; and

storing one or more electronic records, by the computer, comprising the extracted active window, the textual title of the active window, and the user identifier to a data store.

2. A method according to claim 1, wherein identifying the active window comprises:

locating edges of one or more rectangles present in the received screenshot;

identifying one or more rectangles corresponding to the located edges;

detecting one or more windows corresponding to the identified rectangles; and

identifying the active window based on the screen location of the HID input within the detected one or more windows.

3. A method according to claim 1, wherein determining the textual title of the active window comprises:

recognizing characters in a general region of a title bar of the active window using an object character recognition (OCR) technique; and

extracting the characters to develop the textual title of the active window.

4. A method according to claim 1, wherein the HID includes one or more of:

a computer mouse, a keyboard, a touchscreen, and an audio input.

5. A method according to claim 1,

wherein receiving a screenshot of a display device further comprises receiving one or more of:

a device identifier, a network identifier, an active user program identifier, raw HID inputs, and a date/time stamp; and

wherein the storing one or more electronic records further comprises storing one or more of;

the device identifier, the network identifier, the active user program identifier, the raw HID inputs, and the date/time stamp.

6. A method according to claim 1, furthering comprising:

replacing the textual title of the active window with an internal dictionary key.

7. A method according to claim 1, wherein storing the one or more electronic records further comprises:

indexing the textual title of the active window to enable querying by the computer in a GUI environment.

8. A method according to claim 1, further comprising:

retrieving the extracted active window, the textual title of the active window and the user identifier from the data store; and

displaying the extracted active window, the textual title of the active window and the user identifier on a display device of the computer.

9. A method according to claim 1, wherein steps of the method are performed using a set of instructions of a program, wherein the program is an application executed on one or more of:

a cloud computing environment;

a mobile device; and

a desktop device.

10. A method according to claim 1, further comprising:

loading, by the computer, the one or more electronic records; and

displaying, by the computer on a display device of the computer, the loaded one or more electronic records including one or more of;

the extracted active window;

the textual title of the active window; and

the user identifier.

11. A method according to claim 1, further comprising:

searching, by the computer, an index of the one or more electronic records based on a query;

ranking results of the searching; and

displaying the results according to the ranking.

12. A computer program product for auditing activity of a user on a user computing device, the computer program product comprising:

one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions comprising;

program instructions to receive, by a computer, a screenshot of a display device of the user computing device when the user computing device receives an input in a graphical user interface (GUI) from a human interface device (HID), and a screen location of the HID input, and a user identifier;

program instructions to identify, by the computer, an active window in the received screenshot, based on the screen location of the HID input;

program instructions to extract, by the computer, from the screenshot, the identified active window;

program instructions to determine, by the computer, a textual title of the active window; and

program instructions to store one or more electronic records, by the computer, comprising the extracted active window, the textual title of the active window, and the user identifier to a data store.

13. A computer system for auditing activity of a user on a user computing device, the computer system comprising:

one or more computer processors;

one or more computer-readable storage media;

program instructions stored on the computer-readable storage media for execution by at least one of the one or more processors, the program instructions comprising;

program instructions to receive, by a computer, a screenshot of a display device of the user computing device when the user computing device receives an input in a graphical user interface (GUI) from a human interface device (HID), and a screen location of the HID input, and a user identifier;

program instructions to identify, by the computer, an active window in the received screenshot, based on the screen location of the HID input;

program instructions to extract, by the computer, from the screenshot, the identified active window;

program instructions to determine, by the computer, a textual title of the active window; and

program instructions to store one or more electronic records, by the computer, comprising the extracted active window, the textual title of the active window, and the user identifier to a data store.

14. A computer system according to claim 13, wherein program instructions to identify the active window comprises:

program instructions to locate edges of one or more rectangles present in the received screenshot;

program instructions to identify one or more rectangles corresponding to the located edges;

program instructions to detect one or more windows corresponding to the identified rectangles; and

program instructions to identify the active window based on the screen location of the HID input within the detected one or more windows.

15. A computer system according to claim 13, wherein program instructions to determine the textual title of the active window comprises:

program instructions to recognize characters in a general region of a title bar of the active window using an object character recognition (OCR) technique; and

program instructions to extract the characters to develop the textual title of the active window.

16. A computer system according to claim 13,

wherein the program instructions to receive a screenshot of a display device further comprises program instructions to receive one or more of:

a device identifier, a network identifier, an active user program identifier, raw HID inputs, and a date/time stamp; and

wherein the program instructions to store one or more electronic records further comprises program instructions to store one or more of;

the device identifier, the network identifier, the active user program identifier, the raw HID inputs, and the date/time stamp.

17. A computer system according to claim 13, furthering comprising:

program instructions to replace the textual title of the active window with an internal dictionary key.

18. A computer system according to claim 13, further comprising:

program instructions to retrieve the extracted active window, the textual title of the active window and the user identifier from the data store; and

program instructions to display the extracted active window, the textual title of the active window and the user identifier on a GUI display of the computer.

19. A computer system according to claim 13, further comprising:

program instructions to load, by the computer, the one or more electronic records; and

program instruction to display, by the computer on a display device of the computer, the loaded one or more electronic records including one or more of;

the active window;

the textual title of the active window; and

the user identifier of the active window.

20. A computer system according to claim 13, further comprising:

program instructions to search, by the computer, an index of the one or more electronic records based on a query;

program instructions to rank results of the search; and

program instructions to display the results according to the rank.