Remote user interface updates using difference and motion encoding

• US 8,127,233 B2
• Filed: 09/24/2007
• Issued: 02/28/2012
• Est. Priority Date: 09/24/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: McDowell, Brian
• Primary Examiner(s): SAX, STEVEN PAUL
• Application Number:
  US11/860,008
• Publication Number:
  US 20090080523A1
• Time in Patent Office: 1,618 Days
• Field of Search: 345473-475, 345634-641, 345/686, 715720-726, 715744-749, 715762-763, 715/790, 715/802, 715806-807, 375/240.16
• US Class Current: 715/748
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/467 characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation

Abstract

Frames of user interface graphical data can be remotely rendered at a client during a remote session with a server by providing graphical data commands to the client. The commands include motion commands derived from objects that change position between a current frame and a new frame and delta commands derived from differences between the frames. The delta commands can be generated from a frame update after applying motion commands or without applying motion commands. A server identifies moving objects having a first position in the current frame and a second position in the new frame, generates motion hints for the moving objects, and reduces the motion hints based on collision detection, motion verification and other factors. Motion commands are generated for the reduced set of motion hints and applied to a copy of the current frame at the server. Differences between the modified current frame and the new frame are then encoded as delta commands. The server then sends the motion commands and delta commands to the client. The client receives and applies the commands to the current frame to render the new frame.

Claims

1. A method for updating a user interface, comprising:

providing current graphical frame data for a current frame to a client by a server over a network;

identifying a new frame by the server to be rendered at the client;

identifying one or more moving objects having a first position in the current frame and a second position in the new frame;

updating the current frame based on the moving objects, including identifying a first set of moving objects which cannot be moved without colliding with other moving objects, performing a verification process for a motion hint based on differences between source data and destination data for the motion hint, and eliminating motion hints associated with the first set of objects from a subset of motion hints which fail the verification process;

determining a set of difference data associated with the difference between the new frame and the updated current frame;

encoding one or more motion commands based on the moving objects and one or more delta commands based on the difference data into a payload; and

transmitting the payload to the client.

2. The method of claim 1, wherein said step of updating includes:

generating a motion vector for each of the moving objects.

3. The method of claim 2, wherein said step of updating includes:

applying each motion vector to pixel data corresponding to the moving object in the current frame.

4. The method of claim 3, wherein the transmitting the payload includes:

transmitting the motion vectors encoded in the one or more motion commands.

5. The method of claim 4, wherein said step of generating a motion vector includes:

specifying a shape of the object; and

specifying a vector indicative of the object motion.

6. The method of claim 5, wherein the shape is specified as a rectangle with parameters of an origin, height and width.

7. The method of claim 4, wherein said step of generating a motion vector includes:

generating a motion hint for each of the one or more moving objects; and

generating a motion command for the subset of motion of hints.

8. The method of claim 1, wherein said one or more delta commands specify a length of consecutive pixels within a raster of the new image.

9. The method of claim 1, wherein said step of determining a set of difference data includes:

comparing pixel data values of the modified current frame with pixel data value of the new frame for a portion of the new frame determined to have changed with respect to the current frame.

10. The method of claim 4, further comprising:

generating the payload, the payload including data to update the current graphical frame data into modified graphical frame data associated with the new frame, the payload including commands for separately updating two or more types of graphical data formats.

14. The method of claim 1, further comprising:

establishing a remote session between the client and the server over the network, the client is a thin client, the establishing a remote session is performed prior to providing the current graphical frame data.

15. The method of claim 14, wherein the encoding one or more motion commands based on the moving objects and one or more delta commands based on the difference data into a payload includes:

encoding delta data associated with differences in pixel data between the new frame and the updated current frame.

16. The method of claim 15, wherein said step of encoding delta data includes:

generating a set of dirty rectangles indicating a change in pixel values between the current frame and new frame; and

encoding pixel delta data associated with pixels contained within the dirty rectangles.

17. The method of claim 16, wherein said step of generating a set of dirty rectangles includes:

identifying two or more dirty rectangles that overlap; and

replacing the two or more overlapping dirty rectangles with two or more non-overlapping dirty rectangles which encompass the same pixels in the new frame.

18. The method of claim 15, wherein said step of encoding delta data includes:

accessing pixel data for the current frame;

modifying the pixel data for the current frame; and

encoding delta data associated with the difference between the modified pixel data for the current frame and the pixel data for the new frame.

11. A method for updating an interface, comprising:

transmitting a current frame graphical data to a client by a server over a network, the current frame graphical data to be provided through a user interface render engine;

receiving user input from the client;

identifying a new frame to provide to the client by the server in response to the user input;

identifying one or more moving objects having a first position in the current frame and a second position in the new frame;

generating a list of motion operations to be performed on the current frame;

updating the motion operations list by removing one or more motion operations from the operation list which would result in overwriting pixels associated with another motion object;

encoding the updated list of motion operations into a payload;

applying motion operations in the updated operations list to a copy of the current frame at the server to generate a modified current frame;

determining a difference in pixel values between the modified current frame and the new frame;

encoding the pixel value differences between the modified current frame and the new frame as delta commands into the payload;

applying the delta commands to the modified current frame at the server; and

transmitting the payload to the client from the server.

12. The method of claim 11, wherein said step of encoding the pixel value differences includes:

encoding RGB pixel value differences into the payload; and

encoding alpha pixel value differences into the payload.

13. The method of claim 11, wherein said step of updating the motion operations list includes:

comparing pixel data values for pixels within a bounded area associated with a moving object, the moving object associated with one of the motion operations in the motion operations list;

determining results of the comparison for a subset of all bounded area pixels satisfy a threshold;

removing the said motion operation based on said step of determining results.