ENHANCED IMAGE ENCODING IN A VIRTUAL DESKTOP INFRASTRUCTURE ENVIRONMENT

• US 20150215624A1
• Filed: 01/28/2014
• Published: 07/30/2015
• Est. Priority Date: 01/28/2014
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: WEI, Yuping
• Granted Patent: US 9,986,242 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence

Abstract

A method is provided to perform enhanced image encoding in a virtual desktop infrastructure (VDI) environment that includes a client device having a first graphical user interface (GUI) and a server device having a second GUI associated with the first GUI. One example method may include receiving, from the client device, user interface (UI) events based on which the first GUI is to be updated, and obtaining, by an agent on the server device, UI information relating to a UI element on the second GUI. The method may further include determining, by the agent on the server device, an encoding configuration for an encoder based at least in part on the UI information, encoding image data of the second GUI based on the encoding configuration, and sending encoded image data to the client device to update the first GUI on a screen of the client device.

Claims

1. A method to perform enhanced image encoding in a virtual desktop infrastructure (VDI) environment comprising a client device that supports a first graphical user interface (GUI) and a server device that supports a second GUI associated with the first GUI, the method comprising:

receiving, from the client device, user interface (UI) events based on which the first GUI is to be updated;

obtaining, by an agent on the server device, UI information relating to a UI element on the second GUI, the UI information being obtained from an operating system using an application programming interface (API) supported by the operating system on the server device, the UI information indicating updates to the second GUI caused by the received UI events;

determining, by the agent on the server device, an encoding configuration for an encoder supported by the server device based at least in part on the UI information;

encoding, by the encoder, image data of the second GUI using the encoder based on the encoding configuration; and

sending, by the server device, encoded image data to the client device to update the first GUI on a screen of the client device.

2. The method of claim 1, wherein determining the encoding configuration for the encoder comprises:

comparing a current frame and a previous frame of the first GUI to determine a frame change; and

determining the encoding configuration for the encoder based on the frame change and UI information.

3. The method of claim 1, wherein determining the encoding configuration for the encoder based at least in part on the UI information comprises:

when the updates to the second GUI comprise movement of a UI element from a first position to a second position on the second GUI, determining a motion vector associated with the movement of the UI element for the encoder.

4. The method of claim 1, wherein determining the encoding configuration for the encoder based at least in part on the UI information comprises:

when the updates to the second GUI comprise creation of a new UI element or update of a UI element on the second GUI, determining an encoding mode for the encoder.

5. The method of claim 4, wherein determining the encoding mode comprises selecting from one of text encoding mode, inter-frame encoding mode and intra-frame encoding mode.

6. The method of claim 1, wherein determining the encoding configuration for the encoder based at least in part on the UI information comprises:

when the updates to the second GUI comprise creation of a new UI element or update of a UI element on the second GUI, determining reference frame information for the encoder.

7. The method of claim 6, wherein determining the reference frame information comprises saving the current frame as a short-term reference frame or long-term reference frame.

8. The method of claim 1, wherein the VDI environment supports seamless window application in which virtual machine applications provided by the server device appear to operate natively on the first GUI of the client device.

9. The method of claim 1, wherein the framebuffer encoder employs one of the following encoding techniques:

H.264/Advanced Video Coding (AVC), Personal Computer over Internet Protocol (PCoIP), and Portable Network Graphics (PNG).

10. The method of claim 1, wherein:

the UI events received from the client device comprise UI input events associated with a user'"'"'s interaction with the first GUI, the UI input events comprising at least one of keyboard input event, mouse input event, touch screen input event; and

the UI information relating to the UI element on the second GUI comprises at least one of identifier, type, position, size, color, sub-element, font type, text, title, state, and shape of the UI element.

11. The method of claim 1, wherein:

the server device is virtualized into one or more virtual machines;

a guest operating system of one of the one or more virtual machines is configured to support a VDI host agent that receives the UI events, and determines the encoding configuration for the encoder.

12. A non-transitory computer-readable storage medium including a set of instructions which, when executed by a processor, causes the processor to perform an enhanced image encoding method in a virtual desktop infrastructure (VDI) environment comprising a client device that supports a first graphical user interface (GUI) and a server device that supports a second GUI associated with the first GUI, the method comprising:

receiving, from the client device, user interface (UI) events based on which the first GUI is to be updated;

obtaining, by an agent on the server device, UI information relating to a UI element on the second GUI, the UI information being obtained from an operating system using an application programming interface (API) supported by the operating system on the server device, the UI information indicating updates to the second GUI caused by the received UI events;

determining, by the agent on the server device, an encoding configuration for an encoder supported by the server device based at least in part on the UI information;

encoding, by the encoder, image data of the second GUI based on the encoding configuration; and

sending, by the server device, encoded image data to the client device to update the first GUI on a screen of the client device.

13. The non-transitory computer-readable storage medium of claim 12, wherein determining the encoding configuration for the encoder comprises:

comparing a current frame and a previous frame of the first GUI to determine a frame change; and

determining the encoding configuration for the encoder based on the frame change and UI information.

14. The non-transitory computer-readable storage medium of claim 12, wherein determining the encoding configuration for the encoder based at least in part on the UI information comprises:

when updates to the second GUI comprise movement of a UI element from a first position to a second position on the second GUI, determining a motion vector associated with the movement of the UI element.

15. The non-transitory computer-readable storage medium of claim 12, wherein determining the encoding configuration for the encoder based at least in part on the UI information comprises:

when updates to the second GUI comprise creation of a new UI element or update of a UI element on the second GUI, determining an encoding mode for the encoder.

16. The non-transitory computer-readable storage medium of claim 15, wherein the encoding mode is at least one of text encoding mode, inter-frame encoding mode, and intra-frame encoding mode.

17. The non-transitory computer-readable storage medium of claim 12, wherein determining the encoding configuration based at least in part on the UI information comprises:

when the updates to the second GUI comprise creation of a new UI element or update of a UI element on the second GUI, determining reference frame information for the encoder.

18. The non-transitory computer-readable storage medium of claim 17, wherein determining the reference frame information comprises saving the current frame as a short-term reference frame or long-term reference frame.

19. The non-transitory computer-readable storage medium of claim 12, wherein the VDI environment supports seamless window application in which virtual machine applications provided by the server device appear to operate natively on the first GUI of the client device.

20. The non-transitory computer-readable storage medium of claim 12, wherein the framebuffer encoder employs one of the following encoding techniques:

H.264/Advanced Video Coding (AVC), Personal Computer over Internet Protocol (PCoIP), and Portable Network Graphics (PNG).

21. The non-transitory computer-readable storage medium of claim 12, wherein:

the UI events received from the client device comprise UI input events associated with a user'"'"'s interaction with the first GUI, the UI input events comprising at least one of keyboard input event, mouse input event, touch screen input event; and

the UI information relating to the UI element on the second GUI comprises at least one of identifier, type, position, size, color, sub-element, font type, text, title, state, and shape of the UI element.