Loss tolerant protocol for remoting desktop graphics

• US 9,225,784 B2
• Filed: 05/31/2013
• Issued: 12/29/2015
• Est. Priority Date: 01/22/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Schmieder, Wilhelm R., Wynn, Tong L., Abdo, Nadim Y., Fagg, Graham Edward
• Primary Examiner(s): Ibrahim, Mohamed
• Application Number:
  US13/906,962
• Publication Number:
  US 20130268685A1
• Time in Patent Office: 942 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04L 63/06 for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/166 at the transport layer
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/24 Negotiation of communication capabilities
• CPC Class Codes: H04L 69/26 Special purpose or proprietary protocols or architectures network applications for proprietary or special purpose networking environments H04L67/12

Abstract

In various embodiments, remote presentation encoding techniques may be modified in such a way that the data can be transmitted over transports without guaranteed packet delivery. In one embodiment, the desktop graphics data may be encoded in individual frames, each frame comprising self-contained graphics elements that fit in a small number of User Datagram Protocol (UDP) protocol data units (PDUs). The PDUs may then be sent to the client on a separate lossy link instead of the lossless link. On the client side, the client may detect which graphic elements within a frame were “lost” as a result of dropped UDP packets and request a refresh from the server through the lossless channel.

Claims

1. A computer readable storage device storing thereon computer executable instructions for:

establishing a first connection with a remote graphics source, the first connection managing graphics data transmission to the remote graphics source and established using a lossless protocol;

establishing a second connection to send graphics data to the remote graphics source, the second connection established using a lossy protocol;

determining, using the first connection, a status of the second connection; and

managing, using the first connection, the second connection based on the determined status.

2. The computer readable storage device of claim 1, further comprising determining that a portion of said remote graphics data was not received and transmitting, using the first connection, an indication of the portion of data not received.

3. The computer readable storage device of claim 2, wherein said determining comprises waiting for a predetermined period of time after receiving other portions of the remote graphics data to evaluate the status of the second connection.

4. The computer readable storage device of claim 2, wherein said determining comprises:

for each frame, instantiating a memory object tracking regions of a client screen corresponding to the frame;

as the remote presentation graphics data is received, identifying regions of the client screen represented by the received remote presentation graphics data; and

determining that at least a part of the client screen has not been received based on the tracked regions.

5. The computer readable storage device of claim 1, wherein the status of the second connection is determined by evaluating control information managed by the first connection.

6. The computer readable storage device of claim 1, wherein the managing further comprising terminating the second connection and using the first connection to receive the remote presentation graphics data as a fallback.

7. A method for transmitting graphics data from a first computer to a second computer, the method comprising:

causing a first connection to be established between the first computer and the second computer, the first connection managing graphics data transmission between the first computer and the second computer and established using a lossless protocol;

causing to be established a second connection to send graphics data from the first computer to the second computer, the second connection established using a lossy protocol;

determining, using the first connection, a status of the second connection; and

managing, using the first connection, the second connection based on the determined status.

8. The method of claim 7, wherein the managing further comprising terminating the second connection and using the first connection to transmit the graphics data as a fallback transmission method.

9. The method of claim 7, further comprising, using the first connection to provide feedback regarding graphics data that was not received, and in response to the feedback, transmitting, over the second connection, the remote presentation graphics data that was not received.

10. The method of claim 7, wherein the lossless protocol is TCP and the lossy protocol is UDP.

11. The method of claim 7, further comprising compressing the graphics data before transmitting the graphics data, wherein a number of data packets for a compression history for the compressed graphics data is determined as a function of network conditions.

12. The method of claim 7, further comprising:

encrypting the graphics data before transmitting the graphics data; and

using the first connection, transmitting to the second computer an encryption key for decrypting the encrypted graphics data and an indication of a range of data packets for which the encryption key should be used.

13. The method of claim 7, further comprising:

encrypting the graphics data before transmitting the graphics data, wherein the encrypting is accomplished using one of a predetermined number of encryption keys;

transmitting the predetermined number of encryption keys to the client second computer using the first connection; and

transmitting to the client second computer a selection identifying one of the encryption keys to be used for decrypting the graphics data.

14. The method of claim 7, wherein the graphics data is transmitted in data packets and each data packet comprises a unique frame number and a unique sequence number identifying an order within a frame.

15. The method of claim 7, wherein data over the second connection is transmitted in maximum transmission units (MTUs) and a number of packets encoded in one MTU is determined based on a link loss rate.

16. The method of claim 7, wherein the graphics data comprises geometry information that identifies a frame position corresponding to the graphics data.

17. The method of claim 7, wherein the status of the second connection is determined by evaluating control information managed by the first connection.

18. The method of claim 7, further comprising sending duplicate data for a portion of the graphics data that meet a predetermined criterion.

19. A system comprising:

at least one processor; and

at least one memory communicatively coupled to said at least one processor, the memory having stored therein computer-executable instructions that, when executed by the at least one processor, cause the system to;

establish a first connection with a computing device, the first connection managing graphics data to the computing device and established using a lossless protocol;

establish a second connection to send graphics data to the computing device, the second connection established using a lossy protocol;

determine, using the first connection, a status of the second connection; and

manage, using the first connection, the second connection based on the determined status.

20. The system of claim 19, further comprising computer-executable instructions that, when executed by the at least one processor, cause the system to encode the second data content into data representing frames, each frame comprising information for representing a client screen, wherein the second data content is encoded so that the client screen is divided into self-contained graphics elements.