Method and system for image sequence transfer scheduling

• US 9,286,082 B1
• Filed: 05/24/2013
• Issued: 03/15/2016
• Est. Priority Date: 04/06/2005
• Status: Active Grant

• Current Assignee: Teradici Corporation (HP Inc.)
• Original Assignee: Teradici Corporation (HP Inc.)
• Inventors: Hobbs, David Victor
• Primary Examiner(s): Wai, Eric C
• Application Number:
  US13/902,204
• Time in Patent Office: 1,026 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: H04L 67/08 specially adapted for terminal emulation, e.g. Telnet

Abstract

A method and apparatus for generating and communicating encoded images. In one embodiment, the apparatus comprises a graphics processor for rendering, for each display of a plurality of displays, a plurality of images from a plurality of drawing command sets, wherein each drawing command set is generated by a separate VM of a plurality of VMs in response to HID events received from a network location associated with a corresponding display; an image encoder for generating an encoding of each image of the plurality of images, each encoding comprising indications of changed and unchanged portions of a corresponding image; and a scheduler, coupled to the image encoder by a computer expansion bus, for scheduling generation of the encoding based on completion of (i) a rendering of an associated image from the plurality of images and (ii) a transmitting, via the IP network, of an encoding of a previously associated image.

Claims

1. An apparatus for generating and communicating encoded images, comprising:

one or more graphics processors for rendering, for each display of a plurality of displays, a plurality of images from a plurality of drawing command sets;

one or more hardware based encoders that generate an encoding of each image of the plurality of images, wherein each encoding comprises indications of changed and unchanged portions of a corresponding image of the plurality of images;

at least one storage device storing processor-executable instructions which, when executed by one or more processors, coupled to the one or more graphics processors and the one or more hardware based encoders, performs a method comprising;

generating each drawing command set of the plurality of drawing commands sets by executing, on the one or more processors, a separate virtual machine (VM) of a plurality of VMs in response to Human Input Device (HID) events received via an Internet Protocol (IP) network from a network location associated with a corresponding display of the plurality of displays, wherein the plurality of images represent application output of a remote desktop session; and

scheduling generation of the encoding of each image of the plurality of images based on completion of

(i) a rendering of an associated image from the plurality of images and

(ii) a transmitting at an available IP network bandwidth, via the IP network, of an encoding of a previously associated image from the plurality of images, wherein scheduling generation of a first encoding of the plurality of images increases a use of an image communication resource in response to an input event associated with the first encoding and scheduling generation of the first encoding comprises dropping at least one image rendered from the plurality of drawing commands sets.

2. The apparatus of claim 1, wherein the one or more hardware based encoders identifies changed text portions and changed picture portions of each image of the plurality of images, encodes the changed text portions using a lossless encoder, and encodes changed picture portions using a lossy encoder.

3. The apparatus of claim 1, wherein all VMs of the plurality of VMs are executed on the one or more processors coupled to the one or more graphics processors by a computer expansion bus, wherein the one or more graphics processors is an image-rendering engine of a Graphics Processing Unit (GPU), wherein the computer expansion bus is a Peripheral Component Interconnect Express (PCIe) bus, and wherein each of the VMs of the plurality of VMs comprises an operating system that generates an associated drawing command set of the plurality of drawing command sets.

4. The apparatus of claim 1, wherein changed portions of the plurality of images are transferred to the one or more hardware based encoders based on the indications of changed and unchanged portions.

5. The apparatus of claim 1, wherein the one or more hardware based encoders directly accesses the plurality of images from a memory of the one or more graphics processors.

6. The apparatus of claim 1, wherein a signal indicating a completion of a rendering of an image of the plurality of images is delayed until the image is encoded and partly transmitted, and wherein a delay of the signal delays a rendering of subsequent drawing commands for a display associated with the image.

7. The apparatus of claim 1, wherein scheduling generation of the encoding determines a completion of a rendering of a first image of the plurality of images, determines a schedule for communicating an encoding of the first image to a display of the plurality of displays, and delays drawing commands for a subsequent update to the first image based on the schedule for communicating the encoding of the first image.

8. The apparatus of claim 1, wherein notification, to a corresponding VM, of completion of rendering an image of the plurality of images is delayed based on a communication schedule of the image to a display of the plurality of displays.

9. The apparatus of claim 1, wherein the one or more hardware based encoders selectively encodes each portion of a plurality of portions of an image of the plurality of images as one of unchanged, a lossless encoding, or a lossy encoding, based on an analysis of the image.

10. The apparatus of claim 1, wherein the one or more hardware based encoders selects a lossless or a lossy encoding for each changed portion of the plurality of images based on contrast and color content of the changed portion.

11. A method for generating and communicating encoded images, comprising:

rendering, by one or more graphics processors and for each display of a plurality of displays, a plurality of images from a plurality of drawing command sets;

generating, by one or more hardware encoders, an encoding of each image of the plurality of images, wherein each encoding comprises indications of changed and unchanged portions of a corresponding image of the plurality of images;

generating each drawing command set of the plurality of drawing commands sets by executing, on one or more processors, a separate virtual machine (VM) of a plurality of VMs in response to Human Input Device (HID) events received via an Internet Protocol (IP) network from a network location associated with a corresponding display of the plurality of displays, wherein the plurality of images represent application output of a remote desktop session; and

scheduling, by a scheduler executing on the one or more processors, generation of the encoding of each image of the plurality of images based on completion of (i) a rendering of an associated image from the plurality of images and (ii) a transmitting, via the IP network, of a previous encoding of a previously associated image from the plurality of images, wherein scheduling generation of a first encoding of the plurality of images increases a use of an image communication resource in response to an input event associated with the first encoding, and scheduling generation of the first encoding comprises dropping at least one image rendered from the plurality of drawing commands sets in rendering the plurality of images and wherein the one or more processors are coupled to the one or more hardware encoders by a computer expansion bus.

12. The method of claim 11, wherein generating the encoding of each image comprises identifying changed text portions and changed picture portions of each image of the plurality of images, encoding the changed text portions using a lossless encoder, and encoding changed picture portions using a lossy encoder.

13. The method of claim 11, wherein the one or more processors are coupled to the graphics processor by the computer expansion bus, and wherein each of the VMs of the plurality of VMs comprises an operating system that generates an associated drawing command set of the plurality of drawing command sets.

14. The method of claim 11, wherein generating the encoding of each image comprises transferring changed portions of the plurality of images to the image encoder based on the indications of changed and unchanged portions.

15. The method of claim 11, wherein a signal indicating a completion of a rendering of an image of the plurality of images is delayed until the image is encoded and partly transmitted, and wherein a delay of the signal delays a rendering of subsequent drawing commands for a display associated with the image.

16. The method of claim 11, wherein scheduling generation of the first encoding comprises:

determining completion of a rendering of a first image of the plurality of images, determining a schedule for communicating the first encoding of the first image to a display of the plurality of displays, and delaying drawing commands for a subsequent update to the first image based on the schedule for communicating the first encoding of the first image.

17. The method of claim 11, wherein notification, to a corresponding VM, of completion of rendering an image of the plurality of images is delayed based on a communication schedule of the image to a display of the plurality of displays.

18. The method of claim 11, wherein generating the encoding of each image comprises selecting a lossless or a lossy encoding for each changed portion of the plurality of images based on contrast and color content of the changed portion.