REMOTE USER INTERFACE UPDATES USING DIFFERENCE AND MOTION ENCODING

• US 20120133675A1
• Filed: 01/10/2012
• Published: 05/31/2012
• Est. Priority Date: 09/24/2007
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: McDowell, Brian
• Application Number:
  US13/347,643
• Publication Number:
  US 20120133675A1
• Time in Patent Office: Days
• US Class Current: 345/629
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/467 characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation

Abstract

Frames of user interface graphical data can be remotely rendered at a client during a remote session with a server by providing graphical data commands to the client. A server identifies moving objects having a first position in the current frame and a second position in the new frame, generates motion hints for the moving objects, and reduces the motion hints based on collision detection, motion verification and other factors. Motion commands are generated for the reduced set of motion hints and applied to a copy of the current frame at the server. Differences between the modified current frame and the new frame are then encoded as delta commands. The server then sends the motion commands and delta commands to the client. The client receives and applies the commands to the current frame to render the new frame.

Claims

1. A method for updating a user interface, comprising:

establishing a remote session between a thin client and a server over a network;

providing current graphical frame data for a current frame to the thin client by the server;

identifying a new frame by the server to be rendered at the thin client;

encoding delta data associated with differences in pixel data between the new frame and the updated current frame; and

transmitting the set of delta data to the thin client.

2. The method of claim 1, wherein the encoding includes:

separately encoding delta data for each of two or more graphical data formats.

3. The method of claim 1, wherein the encoding includes:

generating a set of dirty rectangles indicating a change in pixel values between the current frame and new frame; and

encoding pixel delta data associated with pixels contained within the dirty rectangles.

4. The method of claim 3, wherein the generating a set of dirty rectangles includes:

identifying two or more dirty rectangles that overlap; and

replacing the two or more overlapping dirty rectangles with two or more non- overlapping dirty rectangles which encompass the same pixels in the new frame.

5. The method of claim 1, wherein the encoding includes:

accessing pixel data for the current frame;

modifying the pixel data for the current frame; and

encoding delta data associated with the difference between the modified pixel data for the current frame and the pixel data for the new frame.

6. A system comprising:

a processor; and

computer readable storage coupled to the processor, the computer readable storage having instructions stored thereon which, when executed on the processor, cause the processor to;

establish a remote session between a thin client and a server over a network;

provide current graphical frame data for a current frame to the thin client by the server;

identify a new frame by the server to be rendered at the thin client;

encode delta data associated with differences in pixel data between the new frame and the updated current frame; and

transmit the set of delta data to the thin client.

7. The system of claim 6, wherein the instructions which cause the processor to encode cause the processor to:

separately encode delta data for each of two or more graphical data formats.

8. The system of claim 6, wherein the instructions which cause the processor to encode cause the processor to:

generate a set of dirty rectangles indicating a change in pixel values between the current frame and new frame; and

encode pixel delta data associated with pixels contained within the dirty rectangles.

9. The system of claim 8, wherein the instructions which cause the processor to generate a set of dirty rectangles cause the processor to:

identify two or more dirty rectangles that overlap; and

replace the two or more overlapping dirty rectangles with two or more non- overlapping dirty rectangles which encompass the same pixels in the new frame.

10. The system of claim 6, wherein the instructions which cause the processor to encode cause the processor to:

access pixel data for the current frame;

modify the pixel data for the current frame; and

encode delta data associated with the difference between the modified pixel data for the current frame and the pixel data for the new frame.

11. A computer readable storage device having instructions stored thereon which, when executed on a processor, cause the processor to perform:

establishing a remote session between a thin client and a server over a network;

providing current graphical frame data for a current frame to the thin client by the server;

identifying a new frame by the server to be rendered at the thin client;

encoding delta data associated with differences in pixel data between the new frame and the updated current frame; and

transmitting the set of delta data to the thin client.

12. The computer readable storage device of claim 11, wherein the encoding includes:

separately encoding delta data for each of two or more graphical data formats.

13. The computer readable storage device of claim 11, wherein the encoding includes:

generating a set of dirty rectangles indicating a change in pixel values between the current frame and new frame; and

encoding pixel delta data associated with pixels contained within the dirty rectangles.

14. The computer readable storage device of claim 13, wherein the generating a set of dirty rectangles includes:

identifying two or more dirty rectangles that overlap; and

replacing the two or more overlapping dirty rectangles with two or more non-overlapping dirty rectangles which encompass the same pixels in the new frame.

15. The computer readable storage device of claim 11, wherein the encoding includes:

accessing pixel data for the current frame;

modifying the pixel data for the current frame; and

encoding delta data associated with the difference between the modified pixel data for the current frame and the pixel data for the new frame.