Manipulating Display Content of a Graphical User Interface

• US 20170352172A1
• Filed: 04/07/2017
• Published: 12/07/2017
• Est. Priority Date: 06/02/2016
• Status: Active Grant

• Current Assignee: Nextlabs
• Original Assignee: Nextlabs
• Inventors: Lim, Keng, Fung, Poon
• Granted Patent: US 11,042,955 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 21/1063 Personalisation
• CPC Class Codes: G06F 21/84 output devices, e.g. displays or monitors
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 1/0021 Image watermarking
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2210/62 Semi-transparency
• CPC Class Codes: G09G 2358/00 Arrangements for display data security
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/397 Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay G09G5/02 takes precedence
• CPC Class Codes: G09G 5/399 using two or more bit-mapped memories, the operations of which are switched in time, e.g. ping-pong buffers

Abstract

A method and system of augmenting display content in a graphical user interface environment. Content produced by a graphical user interface is augmented with additional content before the content is displayed. In an example, a security marker may be rendered on top of an existing display content using the method described to protect high-value or sensitive information.

Claims

1. A method comprises:

providing a computing device having a graphical user interface and a graphics processing unit wherein the graphics processing unit having a video buffer;

providing the graphical user interface having a compositing window manager wherein the compositing window manager produces composite content for a display;

providing a display content augmentation adapter for producing augmentation content wherein the compositing window manager invokes the display content augmentation adapter after producing composite content but before making the video buffer available for video refresh;

opening a document in an application program on the computing device;

producing display content in a display buffer with content of the document by the application program;

producing composite content in the video buffer with the display content in the display buffer by the compositing window manager wherein the composite content comprises content of the document; and

invoking the display content augmentation adapter to render augmentation content over the composite content by the compositing window manager wherein the augmentation content comprises a security marker rendered over content of the document.

2. The method of claim 1 wherein the document is a spreadsheet.

3. The method of claim 1 wherein the document is a 3D CAD drawing.

4. The method of claim 1 wherein the document is a video file.

5. The method of claim 1 wherein the document is a video stream.

6. The method of claim 1 wherein the compositing window manager is Microsoft Windows Desktop Window Manager.

7. The method of claim 1 wherein the display content augmentation adapter is installed in a process during startup phrase of the process.

8. The method of claim 1 wherein the display content comprises rasterized data, a graphical element or a graphics rendering command.

9. The method of claim 8 wherein the graphics rendering command is executed in the graphics processing unit.

10. The method of claim 1 wherein the producing composite content in the video buffer with the display content in the display buffer by the compositing window manager further comprises:

sending a subset of the display content in the display buffer to the graphics processing unit.

11. The method of claim 1 wherein the display content augmentation adapter renders augmentation content using Microsoft Direct3D application programming interface.

12. The method of claim 1 wherein the display content augmentation adapter renders augmentation content using OpenGL application programming interface.

13. The method of claim 1 wherein the augmentation content comprises a text string.

14. The method of claim 1 wherein the augmentation content comprises graphics or an image.

15. The method of claim 1 wherein the augmentation content is semitransparent, wherein the composite content under the augmentation content is visible.

16. The method of claim 1 wherein the augmentation content comprises a disclaimer message.

17. The method of claim 1 wherein the augmentation content comprises a warning message.

18. The method of claim 1 wherein the security marker is opaque, wherein the composite content under the security marker is not visible.

19. The method of claim 1 wherein the security marker describes current user of the computing device.

20. The method of claim 1 wherein the security marker comprises a date.