Remoting desktop displays using move regions

• US 10,031,636 B2
• Filed: 09/08/2011
• Issued: 07/24/2018
• Est. Priority Date: 09/08/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ungureanu, Dorin, Sadek, Mohamed, Schmieder, Robert W., Sidhana, Harneet
• Primary Examiner(s): Repsher, III, John
• Application Number:
  US13/227,491
• Publication Number:
  US 20130067344A1
• Time in Patent Office: 2,511 Days
• Field of Search: 715740, 345557
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]

Abstract

A desktop display over a remote connection is updated by detecting, at their origin, changes in the desktop that reflect movement of an image, such as window scrolls, window moves and other movements of regions. For every frame of the desktop that is composed, moves can be detected. The offsets and regions of these moves are transformed into desktop space. A list of moves and other information about changed regions in the desktop display provide both an indication of moved image regions and move parameters, and image data that has otherwise changed. The moved image regions are moved at the destination by applying the move parameters, without transferring image data for the moved region to the remote device, while the remaining changed portion of the desktop is copied over the connection to the remote device.

Claims

1. A computer-implemented process, performed at a server computer, comprising:

rendering, in memory of the server computer, a desktop composition tree representing display data for a desktop display including at least one surface for an application and a desktop;

processing the desktop composition tree to identify changes in the desktop composition tree resulting in changes in the display data for the desktop display from a first frame to a next frame in a sequence of frames produced for the desktop display over time;

storing the identified changes in a plurality of move arrays, including at least one move array for the at least one surface and a move array for the desktop, the plurality of move arrays storing any move operation involving movement of a region from one position in the desktop display to another position in the desktop display, and further storing dirty regions for the at least one surface and dirty regions for the desktop;

generating, in the memory of the server computer, based on the identified changes, information describing moved regions of the desktop display and move parameters, including for each moved region an indication of a respective moved region and respective parameters of a move operation applied to the respective moved region, the respective parameters including information indicative of an offset of the respective moved region between the first frame and the next frame, and information describing dirty regions of the desktop display, including changed image data for the desktop display other than in the moved regions, by merging the at least one move array for the at least one surface and the move array for the desktop, including merging the dirty regions for the at least one surface and the dirty regions for the desktop; and

sending over a computer network to a remote display device, as display data for the the next frame, the information describing the moved regions of the desktop display and the move parameters, without image data of the desktop display for the moved regions, and the information describing the dirty regions of the desktop display, including the changed image data for the desktop display other than in the moved regions.

2. The computer-implemented process of claim 1, wherein the generating comprises, for each rendering of the desktop, merging the information describing the moved regions of the desktop display and the move parameters with information from a previous rendering of the desktop, wherein the desktop is rendered again prior to sending the information from the previous rendering of the desktop.

3. The computer-implemented process of claim 1, wherein processing the desktop composition tree to identify changes comprises detecting scroll operations applied to the at least one surface, and wherein generating the information describing the moved regions of the desktop display and the move parameters is based on at least the detected scroll operations.

4. The computer-implemented process of claim 1, wherein the desktop composition tree further comprises a plurality of surfaces, wherein each surface of the plurality of surfaces represents an application, and each surface of plurality of surfaces has a respective move array and respective dirty regions.

5. The computer-implemented process of claim 1, wherein the generating comprises, for each rendering of the at least one surface, merging the information describing the moved regions of the desktop display and the move parameters with information from a previous rendering of the at least one surface, wherein the at least one surface is rendered again prior to rendering the desktop display.

6. The computer-implemented process of claim 5, wherein the merging the information describing the moved regions of the desktop display and the move parameters with the information from the previous rendering of the at least one surface comprises:

unioning new dirty regions with existing dirty regions;

for each new dirty region;

in response to a determination that there is a first intersection between a respective new dirty region and a source of a moved region, removing the first intersection from the source of the moved region; and

in response to a determination that there is a second intersection between the respective new dirty region and a destination of the moved region, removing, from the source of the moved region, the second intersection as translated back to the source of the moved region by an offset from move parameters for the moved region.

7. The computer-implemented process of claim 1, wherein the sending over a computer network to the remote display device is accomplished without transferring pixel data for the moved regions of the desktop display to the remote display device.

8. A server computer comprising:

a processor and memory;

a computer storage medium and computer program instructions stored on the computer storage medium which, when processed by the processor, instruct the processor to perform a process comprising;

rendering, in the memory, a desktop composition tree representing display data for a desktop display including at least one surface for an application and a desktop;

processing the desktop composition tree to identify changes in the desktop composition tree resulting in changes in the display data for the desktop display from a first frame to a next frame in a sequence of frames produced for the desktop display over time;

storing, in the memory, the identified changes in a plurality of move arrays, including at least one move array for the at least one surface and a move array for the desktop, the plurality of move arrays storing any move operation involving movement of a region from one position in the desktop display to another position in the desktop display, and further storing dirty regions for the at least one surface and dirty regions for the desktop;

generating, in the memory, based on the identified changes, information describing moved regions of the desktop display and move parameters, including for each moved region an indication of a respective moved region and respective parameters of a move operation applied to the respective moved region, the respective parameters including information indicative of an offset of the respective moved region between the first frame and the next frame, and information describing dirty regions of the desktop display, including changed image data of the desktop display other than in the moved regions, by merging the at least one move array for the at least one surface and the move array for the desktop, including merging the dirty regions for the at least one surface and the dirty regions for the desktop; and

sending over a computer network to a remote display device, as display data for the the next frame, the information describing the moved regions of the desktop display and the move parameters, without image data of the desktop display for the moved regions, and the information describing the dirty regions of the desktop display, including the changed image data of the desktop display other than in the moved regions.

9. The server computer of claim 8, wherein the generating comprises, for each rendering of the desktop, merging the information describing the moved regions of the desktop display and the move parameters with information from a previous rendering of the desktop, wherein the desktop is rendered again prior to sending the information from the previous rendering of the desktop.

10. The server computer of claim 8, wherein processing the desktop composition tree to identify changes comprises detecting scroll operations applied to the at least one surface, and wherein generating the information describing the moved regions of the desktop display and the move parameters is based on at least the detected scroll operations.

11. The server computer of claim 8, wherein the desktop composition tree further comprises a plurality of surfaces, wherein each surface of the plurality of surfaces represents an application, and each surface of the plurality of surfaces has a respective move array and respective dirty regions.

12. The server computer of claim 8, wherein the generating comprises, for each rendering of the at least one surface, merging the information describing the moved regions of the desktop display and the move parameters with information from a previous rendering of the at least one surface, wherein the at least one surface is rendered again prior to rendering the desktop display.

13. The server computer of claim 12, wherein the merging information describing the moved regions of the desktop display and the move parameters with the information from the previous rendering of the surface comprises:

unioning new dirty regions with existing dirty regions;

for each new dirty region;

in response to a determination that there is a first intersection between a respective new dirty region and a source of a moved region, removing the first intersection from the source of the moved region;

in response to a determination that there is a second intersection between the respective new dirty region and a destination of the moved region, removing, from the source of the moved region, the second intersection as translated back to the source of the moved region an offset from move parameters for the moved region.

14. A computing machine including a processor and a computer storage medium storing computer program instructions, the computing machine, when the computer program instructions are processed by the processor, comprising:

a desktop rendering engine having an input for receiving a desktop composition tree including information describing a composition of one or more surfaces for one or more applications and a desktop for a desktop display, and an output providing a sequence of frames as a rendered desktop display over time;

wherein the desktop rendering engine is configured to process the desktop composition tree to identify changes in the desktop composition tree resulting in changes in the display data for the desktop display from a first frame to a next frame in a sequence of frames produced for the desktop display over time, and to store the identified changes in a plurality of move arrays, including at least one move array for the at least one surface and a move array for the desktop, the plurality of move arrays storing any move operation involving movement of a region from one position in the desktop display to another position in the desktop display, and further storing dirty regions for the at least one surface and dirty regions for the desktop;

wherein the desktop rendering engine is configured to generate, based on the detected changes, information describing moved regions within the rendered desktop display and move parameters, including for each moved region an indication of a respective moved region and respective parameters of a move operation applied to the respective moved region, the respective parameters including information indicative of an offset of the respective moved region between the first frame and the next frame, and to generate information describing dirty regions of the desktop display including changed image data for the desktop display other than in the moved regions by merging the at least one move array for the at least one surface and the move array for the desktop, including merging the dirty regions for the at least one surface and the dirty regions for the desktop; and

a remoting server that sends over a computer network to a remote display device, as display data for the next frame, the information describing the moved regions within the rendered desktop display and the move parameters, without image data of the rendered desktop display for the moved regions, and the information describing the dirty regions of the desktop display including the changed image data for the desktop display other than in the moved regions.

15. The computing machine of claim 14, wherein the desktop rendering engine is further configured to, for each rendering of the desktop, merge the information describing the moved regions within the rendered desktop display and the move parameters with move information from a previous rendering of the desktop, wherein the desktop rendering engine is further configured to render the desktop again prior to sending the move information from the previous rendering of the desktop.

16. The computing machine of claim 14, wherein, to process the desktop composition tree to identify changes, the desktop rendering engine is further configured to detect scroll operations applied to the at least one surface, and wherein the desktop rendering engine is configured to generate the information describing the moved regions within the rendered desktop display and the move parameters based on at least the detected scroll operations.