REMOTE COMPUTING PLATFORMS PROVIDING HIGH-FIDELITY DISPLAY AND INTERACTIVITY FOR CLIENTS

• US 20100111410A1
• Filed: 03/20/2009
• Published: 05/06/2010
• Est. Priority Date: 10/30/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Shipeng, Lu, Yan, Wu, Feng, Shen, Huifeng
• Granted Patent: US 8,254,704 B2
• Time in Patent Office: Days
• US Class Current: 382/166
• CPC Class Codes: G06F 3/1415 with means for detecting differences between the image stored in the host and the images displayed on the displays
• CPC Class Codes: G06F 3/1462 with means for detecting differences between the image stored in the host and the images displayed on the remote displays
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2360/122 Tiling
• CPC Class Codes: G09G 2360/18 Use of a frame buffer in a display terminal, inclusive of the display panel
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]

Abstract

A “Remote Display Generator,” as described herein, provides various techniques for providing high-fidelity displays with highly responsive interactive application experiences to clients across a wide range of network bandwidths for remotely hosted applications. In general, the Remote Display Generator uses a compression-friendly remote display architecture as a core. With this compression architecture, actual screen data from a remote server is read out from the display buffer frame by frame, and then compressed with a unified screen codec. Other technologies, including timer-driven screen update models and adaptive transmission mechanisms, are then integrated with various embodiments of the Remote Display Generator to improve overall user experience by improving display quality and responsiveness to user interaction with remotely hosted applications.

Claims

1. A method for displaying screen images of a remotely hosted application on a client display device, comprising steps for:

establishing network connections between one or more clients and a remote host;

initiating one or more applications on the remote host for each client;

using the remote host, generating a current screen image for each client for each corresponding application;

using the remote host, dividing each current screen image into a set of non-overlapping blocks of pixels;

using the remote host, comparing the blocks of pixels of each current screen image to corresponding blocks of pixels from an immediately previous screen image for each client to identify unchanged blocks of pixels in each current screen image;

using the remote host, compressing all unchanged blocks of pixels of each current screen image;

using the remote host, encoding the compressed blocks of pixels for each current screen image;

transmitting the encoded compressed blocks for the current screen image for each corresponding application to each corresponding client;

using each client, decoding the encoded compressed blocks for each received current screen image;

using each client, decompressing the decoded blocks of pixels and reconstructing each received current screen image; and

using each client, displaying the reconstructed current screen images received by each of those clients.

2. The method of claim 1 wherein compressing all unchanged blocks of pixels of each current screen image further comprising steps for:

evaluating each unchanged block of pixels and classifying each of those blocks as a text-type block or as an image-type block; and

applying a different block-specific compression to each block type to compress the unchanged blocks of pixels.

3. The method of claim 2 wherein the block-specific compression applied to text-type block includes steps for:

evaluating text-type blocks to identify a small number of dominant pixel colors for the pixels in the text-type block; and

quantizing the pixels in the text-type block to the small number of dominant pixel colors.

4. The method of claim 1 further comprising steps for setting a separate predefined screen update time for each client as a function of available client bandwidth.

5. The method of claim 1 further comprising steps for:

locally monitoring client interactions with the screen image; and

reporting the monitored client interactions to the host.

6. The method of claim 5 wherein subsequent current screen images transmitted to each client from the host are rendered by the host to reflect the monitored client interactions reported to the host.

7. The method of claim 1 further comprising steps for allowing any client to send a request message to the host to cause the host to resize the current screen image to fit within an available display area on the requesting client.

8. The method of claim 1 further wherein identifying unchanged blocks of pixels in each current screen image further comprises steps for generating a “

skip map”

that indicates the specific blocks of pixels that have not changed for each corresponding current screen image, and wherein each corresponding skip map is transmitted to each corresponding client.

9. The method of claim 8 wherein each client uses any received skip maps to select corresponding unchanged blocks of pixels from a local copy of each an immediately previous screen image for use in reconstructing each received current screen image.

10. A system for generating screen images of a remotely hosted application on a client display device, comprising using a remote server for:

executing an application and rendering a current screen image of the application;

dividing the current screen image into a set of non-overlapping pixel blocks of equal size;

comparing the pixel blocks of the current screen image to corresponding pixel blocks of a buffered copy of a previous screen image for the application;

creating a map of pixel blocks that are unchanged from the previous screen image to the current screen image;

compressing all unchanged pixel blocks of the current screen image;

encoding the compressed pixel blocks and the map of unchanged pixel blocks into a composite bitstream; and

transmitting the composite bitstream to the client.

11. The system of claim 10 further comprising using the client to decode, decompress, render and display the current screen image on a client display device.

12. The system of claim 11 wherein rendering the current screen image further comprises using the map of unchanged pixel blocks to select corresponding unchanged pixel blocks from a local copy of the previous screen image and combining those unchanged pixel blocks with the decompressed pixel blocks recovered from the composite bitstream to render the current screen image.

13. The system of claim 10 wherein compressing all unchanged pixel blocks of the current screen image further comprises first evaluating each unchanged pixel block and classifying each of those blocks as a text-type block or as an image-type block.

14. The system of claim 13 wherein text-type pixel blocks are compressed by identifying a small number of the most dominant pixel colors in the text-type block and quantizing the pixels in the text-type block to those dominant pixel colors.

15. The system of claim 10 further comprising locally monitoring client interactions with the current screen image and reporting the monitored client interactions to the server.

16. The system of claim 15 wherein subsequent current screen images transmitted to the client are rendered by the server to reflect the monitored client interactions reported to the server.

17. A computer-readable medium having computer executable instructions stored therein for displaying remotely hosted applications on a local client, said instructions comprising:

executing an application and rendering a current screen image of the application;

dividing the current screen image into a set of non-overlapping pixel blocks of equal size;

comparing the pixel blocks of the current screen image to corresponding pixel blocks of a buffered copy of a previous screen image for the application;

creating a map of pixel blocks that are unchanged from the previous screen image to the current screen image;

compressing all unchanged pixel blocks of the current screen image;

encoding the compressed pixel blocks and the map of unchanged pixel blocks into a composite bitstream; and

transmitting the composite bitstream to the client.

18. The computer-readable medium of claim 17 further comprising computer-executable instructions for using the client to decode, decompress, render and display the current screen image on a client display device.

19. The computer-readable medium of claim 18 wherein rendering the current screen image further comprises using the map of unchanged pixel blocks to select corresponding unchanged pixel blocks from a local copy of the previous screen image and combining those unchanged pixel blocks with the decompressed pixel blocks recovered from the composite bitstream to render the current screen image.

20. The computer-readable medium of claim 17 wherein compressing all unchanged pixel blocks of the current screen image further comprises:

first evaluating each unchanged pixel block and classifying each of those blocks as a text-type block or as an image-type block; and

compressing text-type pixel blocks by identifying a small number of the most dominant pixel colors in the text-type block and quantizing the pixels in the text-type block to those dominant pixel colors.