Systems, methods, and apparatuses for implementing frame aggregation with screen sharing

• US 9,185,149 B2
• Filed: 03/15/2013
• Issued: 11/10/2015
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Spencer, Barry
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): Woo, Andrew
• Application Number:
  US13/840,204
• Publication Number:
  US 20130346499A1
• Time in Patent Office: 970 Days
• Field of Search: 709/204, 709/207
• US Class Current: 1/1
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

In accordance with disclosed embodiments, there are provided methods, systems, and apparatuses for implementing frame aggregation with screen sharing including, for example, means for receiving, at a server, a stream of delta frames from a publishing client as part of a screen sharing session with one or more viewing clients; establishing a FIFO buffer for each of the respective one or more viewing clients on 1:1 basis; queuing a copy of the stream of delta frames into each of the FIFO buffers corresponding to the one or more viewing clients, wherein the stream of delta frames are transmitted from the respective FIFO buffers to the corresponding one or more client viewers; monitoring each of the respective FIFO buffers for each of the one or more viewing clients to determine if two or more delta frames are concurrently queued in any single one of the respective FIFO buffers at any given time; aggregating the two or more delta frames into a single aggregated delta frame; re-queuing the aggregated delta frame; and transmitting the aggregated delta frame to the respective viewing client. Other related embodiments are disclosed.

Claims

1. A method comprising:

receiving, at a server, a stream of delta frames from a publishing client as part of a screen sharing session with one or more viewing clients;

establishing a FIFO buffer for each of the respective one or more viewing clients on 1;

1 basis;

queuing a copy of the stream of delta frames into each of the FIFO buffers corresponding to the one or more viewing clients, wherein the stream of delta frames are transmitted from the respective FIFO buffers to the corresponding one or more client viewers;

monitoring each of the respective FIFO buffers for each of the one or more viewing clients to determine if two or more delta frames are concurrently queued in any single one of the respective FIFO buffers at any given time;

aggregating the two or more delta frames into a single aggregated delta frame;

re-queuing the aggregated delta frame, wherein re-queuing the aggregated delta frame comprises queuing the aggregated delta frame in the respective FIFO buffer with a priority according to an original position of the two or more delta frames aggregated; and

transmitting the aggregated delta frame to the respective viewing client.

2. The method of claim 1, further comprising:

receiving, at the server, a key frame from the publishing client sharing its screen as a first frame of the stream with the plurality of delta frames following the key frame, wherein the key frame defines the screen of the publishing client in its entirety at the beginning of a screen sharing session with the one or more viewing clients.

3. The method of claim 2, wherein each of the plurality of delta frames from the publishing client represent a subset of the screen of the publishing client corresponding to changes at the screen of the publishing client since a preceding delta frame or since the preceding key frame when the delta frame is the first delta frame of the screen sharing session from the publishing client after the key frame.

4. The method of claim 1, wherein each of the plurality of delta frames from the publishing client specifies changes to the screen of the publishing client and includes less than all of the data required to define the screen of the publishing client in its entirety.

5. The method of claim 1, wherein the method constitutes a screen sharing service as a cloud service over a public Internet to the publishing client and the one or more viewing clients to distribute a video stream of the screen sharing session on behalf of the publishing client.

6. The method of claim 1, further comprising:

establishing a server transmitter for each of the respective one or more viewing clients on 1;

1 basis; and

wherein each server transmitter is dedicated to transmitting queued delta frames for exactly one of the viewing clients from the corresponding FIFO buffer established for the same viewing client.

7. The method of claim 6, wherein transmitting the aggregated delta frame to the respective viewing client comprises transmitting the aggregated delta frame to the respective viewing client from the corresponding FIFO buffer as the delta frames are queued into the FIFO buffer at the greater of:

(a) an unrestricted bandwidth rate at which the delta frames are received into the queue or (b) a restricted bandwidth rate at which each respective viewing client is capable of consuming the delta frames from the server.

8. The method of claim 1, wherein the two or more delta frames become concurrently queued in the FIFO buffers for one of the viewing clients when the viewing client consumes the stream of delta frames from the server slower than the publishing client sends the stream of delta frames to the server.

9. The method of claim 1:

wherein the publishing client sends the stream of delta frames to the server at a rate greater than one of the viewing clients consumes the stream of delta frames from the server; and

wherein aggregating the two or more delta frames into a single aggregated delta frame reduces the quantity of delta frames transmitted to the viewing client and reduces the total payload data transmitted to the viewing client via the stream of delta frames.

10. The method of claim 1:

wherein the server receives the stream of delta frames from the publishing client at a first bandwidth rate;

wherein a first one of the viewing clients receives the stream of delta frames from the server at a second bandwidth rate lesser than the first bandwidth rate via frame aggregation by the server iteratively combine multiple of the delta frames in the stream into aggregated delta frames on behalf of the first one of the viewing clients; and

wherein a second one of the viewing clients receives the stream of delta frames from the server at the first bandwidth rate equal to the rate at which the server receives the stream of delta frames from the publishing client, the second one of the viewing clients to receive the stream of delta frames from the server without frame aggregation on the stream of delta frames by the server.

11. The method of claim 1, further comprising:

sequencing the plurality of delta frames from the publishing client according to a frame identifier of each of the plurality of delta frames; and

queuing the plurality of delta frames into the respective FIFO buffers according to their sequencing regardless of the order in which the plurality of delta frames are received at the server.

12. The method of claim 11, wherein aggregating the two or more delta frames into a single aggregated delta frame comprises:

identifying two or more sequential and adjacent delta frames concurrently queued in one of the respective FIFO buffers awaiting transmission to corresponding viewing client;

retrieving the two or more sequential and adjacent delta frames;

aggregating the two or more sequential and adjacent delta frames by overwriting overlapping areas of the publishing client'"'"'s screen as specified by earlier of the two or more sequential and adjacent delta frames with corresponding overlapping areas as specified by later of the two or more sequential and adjacent delta frames.

13. The method of claim 1, further comprising:

establishing a communications interface with the publishing client prior to receiving the stream from the publishing client beginning with a key frame followed by the plurality of delta frames;

establishing a communications interface with the one or more viewing clients to share the screen from the publishing client prior to receiving the key frame from the publishing client; and

transmitting the stream to the one or more viewing clients beginning with the key frame followed by the plurality of delta frames.

14. The method of claim 1, further comprising:

receiving a first of the plurality of delta frames from the publishing client at time_s;

receiving a second of the plurality of delta frames from the publishing client at time₂;

aggregating the first and the second of the plurality of delta frames into the single aggregated delta frame representing all changes at the presenting client'"'"'s screen from time₁ to time₂; and

transmitting the single aggregated delta frame for time₁ to time₂ to one of the client viewers.

15. The method of claim 1, wherein the server is hosted within a host organization operating remote from the publishing client and remote from the one or more viewing clients.

16. The method of claim 15, wherein the host organization provides screen sharing services as a cloud service over a public Internet to the publishing client and to the one or more viewing clients.

17. Non-transitory computer readable storage media having instructions stored thereon that, when executed by a processor in a server, the instructions cause the server to perform operations comprising:

receiving, at a server, a stream of delta frames from a publishing client as part of a screen sharing session with one or more viewing clients;

establishing a FIFO buffer for each of the respective one or more viewing clients on 1;

1 basis;

queuing a copy of the stream of delta frames into each of the FIFO buffers corresponding to the one or more viewing clients, wherein the stream of delta frames are transmitted from the respective FIFO buffers to the corresponding one or more client viewers;

monitoring each of the respective FIFO buffers for each of the one or more viewing clients to determine if two or more delta frames are concurrently queued in any single one of the respective FIFO buffers at any given time;

aggregating the two or more delta frames into a single aggregated delta frame;

re-queuing the aggregated delta frame, wherein re-queuing the aggregated delta frame comprises queuing the aggregated delta frame in the respective FIFO buffer with a priority according to an original position of the two or more delta frames aggregated; and

transmitting the aggregated delta frame to the respective viewing client.

18. The non-transitory computer readable storage media of claim 17, wherein the operations further comprise:

receiving a first of the plurality of delta frames from the publishing client at time₁;

receiving a second of the plurality of delta frames from the publishing client at time₂;

aggregating the first and the second of the plurality of delta frames into the single aggregated delta frame representing all changes at the presenting client'"'"'s screen from time₁ to time₂; and

transmitting the single aggregated delta frame for time₁ to time₂ to one of the client viewers.

19. The non-transitory computer readable storage media of claim 17:

wherein the publishing client sends the stream of delta frames to the server at a rate greater than one of the viewing clients consumes the stream of delta frames from the server; and

wherein aggregating the two or more delta frames into a single aggregated delta frame reduces the quantity of delta frames transmitted to the viewing client and reduces the total payload data transmitted to the viewing client via the stream of delta frames.

20. A server comprising:

a processor to execute a screen sharing service on behalf of a plurality of clients;

a receive interface of the server to receive a stream of delta frames from a publishing client as part of a screen sharing session with one or more viewing clients;

a FIFO buffer for each of the respective one or more viewing clients on 1;

1 basis;

the screen sharing service to queue a copy of the stream of delta frames into each of the FIFO buffers corresponding to the one or more viewing clients, wherein the stream of delta frames are to be transmitted from the respective FIFO buffers to the corresponding one or more client viewers;

a frame aggregator to monitor each of the respective FIFO buffers for each of the one or more viewing clients to determine if two or more delta frames are concurrently queued in any single one of the respective FIFO buffers at any given time;

the frame aggregator to aggregate the two or more delta frames into a single aggregated delta frame;

the FIFO buffer to requeue the aggregated delta frame by queuing the aggregated delta frame in the respective FIFO buffer with a priority according to an original position of the two or more delta frames aggregated; and

a transmitter to transmit the aggregated delta frame to the respective viewing client.

21. The server of claim 20, further comprising:

a server transmitter for each of the respective one or more viewing clients on 1;

1 basis, wherein each server transmitter is dedicated to transmitting queued delta frames for exactly one of the viewing clients from the corresponding FIFO buffer established for the same viewing client; and

wherein the transmitter to transmit the aggregated delta frame to the respective viewing client comprises the transmitter to transmit the aggregated delta frame to the respective viewing client from the corresponding FIFO buffer as the delta frames are queued into the FIFO buffer at the greater of;

(a) an unrestricted bandwidth rate at which the delta frames are received into the queue or (b) a restricted bandwidth rate at which each respective viewing client is capable of consuming the delta frames from the server.

22. The server of claim 20, wherein the server is hosted within a host organization operating remote from the publishing client and remote from the one or more viewing clients.

23. The server of claim 22, wherein the host organization provides the screen sharing service as a cloud service over a public Internet to the publishing client and the one or more viewing clients to distribute a video stream of the screen sharing session on behalf of the publishing client.