Composition control method for remote application delivery

• US 9,565,227 B1
• Filed: 06/16/2014
• Issued: 02/07/2017
• Est. Priority Date: 06/16/2014
• Status: Active Grant

• Current Assignee: Teradici Corporation (HP Inc.)
• Original Assignee: Teradici Corporation (HP Inc.)
• Inventors: Helter, Paul Andrew, Hobbs, David Victor
• Primary Examiner(s): Baturay, Alicia
• Application Number:
  US14/306,084
• Time in Patent Office: 967 Days
• Field of Search: 709/217
• US Class Current: 1/1
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2360/18 Use of a frame buffer in a display terminal, inclusive of the display panel
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/01 Protocols

Abstract

A method for communicating an image stream, one embodiment comprising negotiating, between a server and client, an MTCC identifying: a window as a portion of a display of the client, and an occluded portion of the window defined by the client; generating, by the server, complying with the MTCC, a sequence of image frames each comprising pixel data in a portion of the frame identified by the occluded portion; transmitting an encoding of a first frame of the sequence not including an encoding of the pixel data in the portion of the first frame; identifying, by the server, an event associated with the sequence; and transmitting, before communication between the server and the client regarding the event, by the server, information of the event in conjunction with an encoding of a second frame of the sequence including an encoding of the pixel data in the portion of the second frame.

Claims

1. A method for communicating an image stream, comprising:

negotiating, by a server, with a client computer, via a network session, a media transport and composition configuration (MTCC) by exchanging co-ordinates, sizing and priorities of display windows to identify (i) a window as a portion of a display of the client computer and (ii) an occluded portion of the window, wherein occlusion of the occluded portion is controlled by the client computer;

generating, by the server and in compliance with the MTCC, a sequence of image frames, wherein each image frame of the sequence of image frames comprises pixel data in a portion of the image frame identified by the occluded portion;

transmitting, via the network session, an encoding of a first frame of the sequence of image frames, wherein the encoding of the first frame does not include an encoding of the pixel data in the portion of the first frame identified by the occluded portion;

identifying, by the server, an event associated with the sequence of image frames; and

transmitting, by the server to the client computer, information of the event in conjunction with an encoding of a second frame of the sequence of image frames to reduce event response latency, wherein the encoding of the second frame includes an encoding of the pixel data in the portion of the second frame identified by the occluded portion, wherein transmitting the information of the event occurs prior to any other communication between the server and the client computer regarding the event, and wherein the client computer composes and displays a display of the sequence of image frames according to the information of the event; and

receiving, at the server, an acknowledgement of the event, from the client after a round trip delay,

wherein transmitting the information of the event in conjunction with the encoding of a second frame and prior to the reception of the acknowledgement at the server reduces a displayed response latency to the event by the client computer by at least the round trip delay as compared to transmitting an encoding of the occluded portion of the second frame subsequent to receiving the acknowledgement at the server.

2. The method of claim 1, wherein negotiating the MTCC comprises the client computer allocating the window, exclusive of the occluded portion, to the server;

and wherein transmitting the encoding of the second frame is initiated, (i) in response to identifying the event and (ii) prior to the server receiving an updated allocation of the display inclusive of the occluded portion.

3. The method of claim 2, further comprising:

receiving, by the server, a response to transmitting the information; and

transmitting, to the client computer, an encoding of a third frame of the sequence of image frames, wherein the encoding of the third frame selectively includes, based on the response, an encoding of the pixel data of the third frame.

4. The method of claim 1, wherein the information comprises an update to media transport and composition (MTC) parameters of the window;

and wherein the client computer determines, based on the information and MTC parameters of a second window, a display of a decoding of the encoding of the second frame, wherein negotiating the MTCC is based on the MTC parameters of the window.

5. The method of claim 1, wherein negotiating the MTCC comprises negotiating an image size of the window and wherein the sequence of image frames is rendered to the image size.

6. The method of claim 1, wherein the event is an upstream event received by the server and wherein the server is required to pass the information of the event downstream to the client computer to update the MTCC.

7. The method of claim 1, wherein the portion of the display is a sub-portion of an overall display dimension of the display.

8. The method of claim 1, wherein the event is a request, from an application rendering the sequence of image frames, to change to a full screen mode, and wherein the information comprises a change of the window to a visible image size of the display.

9. The method of claim 1, wherein the event is at least one of:

a user interaction;

a defined change in an audio stream associated with the sequence of image frames;

a user input provided with content of the sequence of image frames;

or

a change in an application generating the sequence of image frames, the change in the application comprising at least one of;

a completion of a user-initiated activity;

a detection of a user-defined data value;

a rendering of a dialog box;

or

an event, from the application generating the sequence of image frames, in response to a user input received from the client computer.

10. The method of claim 1, wherein (a) the server presents itself as an media transport and composition (MTC) client to an upstream server, and (b) the upstream server renders, encodes and transmits an encoding of an image sequence;

and wherein generating the sequence of image frames comprises decoding the encoding of the image sequence.

11. The method of claim 1, wherein during a period of defined bandwidth utilization of an available bandwidth of the network session, the client computer transmits an initial quality encoding of a frame of the sequence of image frames and wherein the client computer displays a decoding of the initial quality encoding upon receiving the information of the event and prior to completing a decoding of the encoding of the second frame.

12. The method of claim 1, wherein negotiating the MTCC comprises the client computer referencing a prior MTCC resulting from a user interaction.

13. The method of claim 1, wherein negotiating the MTCC comprises the server adjusting and communicating first media transport and composition (MTC) parameters, based on second MTC parameters, wherein first MTC parameters are associated with the server, and the second MTC parameters are received from the client computer and associated with a second server.

14. The method of claim 1, wherein transmitting the encoding of the first frame is based on the MTCC;

wherein the server identifies the event subsequent to transmitting the encoding of the first frame; and

wherein transmitting the information is based on the MTCC and the event.