Remote user interface raster segment motion detection and encoding

• US 8,121,423 B2
• Filed: 10/12/2007
• Issued: 02/21/2012
• Est. Priority Date: 10/12/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: McDowell, Brian
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US11/871,719
• Publication Number:
  US 20090097751A1
• Time in Patent Office: 1,593 Days
• Field of Search: 382/107, 382/181, 382/218, 382/236, 375/240.16
• US Class Current: 382/236
• CPC Class Codes: G06F 15/177 Initialisation or configuration control processor initialisation G06F9/4405
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

Raster segment commands are generated by a delta encoder and are encoded for raster segments detected to have moved in the current frame with respect to a location of the raster segment in a previous frame. Raster segment motion commands are stored in a queue and, when written, copy a raster segment into the current frame from a previous payload or cache. When written from the queue to a payload, raster segment motion commands are applied to a copy of the current frame maintained at the server using a copy of the previous payload sent. When copying a raster segment from a raster segment cache, a raster segment cache command retrieves identified raster segment data from the raster segment cache and copies the raster segment into a particular location in the current frame at the client. The raster segment commands are sent to a client via a payload to copy a raster segment from a previous payload into the current frame at the client or from a raster segment cache to the current frame.

Claims

1. A method comprising:

generating a first set of raster segments from differences between a first frame and a second frame for a user interface;

accessing a second set of raster segments from differences between the second frame and a third frame for the user interface;

identifying a first raster segment of the first set that matches a second raster segment of the second set; and

generating a command to move the first raster segment from a first position in the second frame to a second position in the first frame.

2. The method of claim 1, wherein said step of generating a first set of raster segments includes:

comparing a current frame provided to a user through the user interface at a remote client to a new frame to be provided to the user through the user interface; and

identifying one or more adjacent pixel positions having different values between the current frame and the new frame, the first frame comprising the new frame and the second frame comprising the current frame.

3. The method of claim 2, wherein said step of identifying includes:

determining whether the first raster segment and second raster segment are within the same raster.

4. The method of claim 2, wherein said step of identifying includes:

determining the first raster segment and second raster segment are within different rasters.

5. The method of claim 3, wherein said step of generating a command includes:

caching a command to move the first raster; and

encoding the command to a payload.

6. The method of claim 3, wherein said step of comparing includes:

determining if the first raster segment has data values within a threshold of the second raster segment data values.

7. The method of claim 6, wherein the threshold is applied to the difference between each pair of pixels.

8. The method of claim 6, wherein the threshold is applied to the average pixel difference for each pair of compared segments.

9. The method of claim 8, wherein said first raster and second raster contain RGB data values.

10. The method of claim 1, wherein generating a command includes:

caching a raster segment cache command to move a cached raster segment; and

encoding the raster segment cache command to a payload.

11. The method of claim 10, wherein generating a command includes:

increasing a relevance value for the cached raster segment.

12. The method of claim 1, wherein generating a command includes:

determining a raster segment should be removed from a raster segment cache; and

generating a raster segment cache removal command.

13. The method of claim 1, wherein generating a command includes:

determining a raster segment should be added to a raster segment cache; and

generating a raster segment cache add command.

14. A method comprising:

generating a first set of raster segments from differences between a first frame and a second frame for a user interface;

accessing a second set of raster segments from differences between the second frame and a third frame for the user interface;

determining whether the first raster segment of the first set is within the same raster as a second raster segment of the second set;

determining whether the data values for the first raster segment are within a threshold of the data values of the second raster segment;

creating a command to move the first raster segment from a first position in the second frame to a second position in the first frame based on said steps of determining;

queuing the command at a server;

encoding the queued command to a payload; and

transmitting the payload to a client.

15. The method of claim 14, further comprising:

determining whether the first raster segment matches a raster segment contained in a raster segment cache, wherein the second set of raster segments is contained in the raster segment cache stored at the server which contains the same cached raster segments as a corresponding raster segment cache stored at the client which receives the payload;

generating a raster segment cache command to move a selected cached raster segment that corresponds to the first raster segment;

increasing the relevance of the raster segment identified in the raster segment cache command; and

encoding the raster segment cache command to a payload.

16. The method of claim 14, wherein the threshold is applied to each pixel difference between each pair of corresponding pixel values in a pair of compared segments.

17. The method of claim 14, wherein the threshold is applied to the average pixel difference in a pair of compared segments.

18. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising:

generating a first set of raster segments from differences between a new frame to be rendered at a remote client having a user interface and a current frame rendered at the remote client;

accessing a second set of raster segments from differences between the current frame and a previous frame rendered at the remote client;

identifying a first raster segment of the first set that matches a second raster segment of the second set;

generating a raster segment cache command to move the first raster segment from a first position in the current frame to a second position in the new frame; and

providing the raster segment cache command to the remote client.

19. The one or more processor readable storage devices of claim 18, wherein the generating a first set of raster segments includes:

identifying one or more adjacent pixel positions having different values between the current frame and the new frame.