Systems, methods, and apparatuses for implementing frame aggregation with screen sharing

• US 10,732,917 B2
• Filed: 07/17/2018
• Issued: 08/04/2020
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Spencer, Barry L.
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): Woo, Andrew
• Application Number:
  US16/037,896
• Publication Number:
  US 20190034149A1
• Time in Patent Office: 749 Days
• Field of Search: 709204
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

In accordance with disclosed embodiments, there are provided methods, systems, and apparatuses for implementing frame aggregation with screen sharing including, for example, means for receiving, at a server, a stream of delta frames from a publishing client as part of a screen sharing session with one or more viewing clients; establishing a FIFO buffer for each of the respective one or more viewing clients on 1:1 basis; queuing a copy of the stream of delta frames into each of the FIFO buffers corresponding to the one or more viewing clients, wherein the stream of delta frames are transmitted from the respective FIFO buffers to the corresponding one or more client viewers; monitoring each of the respective FIFO buffers for each of the one or more viewing clients to determine if two or more delta frames are concurrently queued in any single one of the respective FIFO buffers at any given time; aggregating the two or more delta frames into a single aggregated delta frame; re-queuing the aggregated delta frame; and transmitting the aggregated delta frame to the respective viewing client. Other related embodiments are disclosed.

Claims

1. A method comprising:

receiving, at a server, a stream of delta frames from a publishing client as part of a screen sharing session with one or more viewing clients;

establishing a FIFO buffer for each of the respective one or more viewing clients on 1;

1 basis;

queuing a copy of the stream of delta frames into each of the FIFO buffers corresponding to the respective one or more viewing clients, wherein the respective copy of the stream of delta frames are transmitted from the respective FIFO buffers to the corresponding one or more viewing clients;

monitoring each of the respective FIFO buffers for each of the respective one or more viewing clients to determine if two or more delta frames are concurrently queued in any single one of the respective FIFO buffers at any given time;

aggregating the two or more delta frames into a single aggregated delta frame;

re-queuing the single aggregated delta frame; and

transmitting the single aggregated delta frame to the respective one or more viewing clients.

2. The method of claim 1, further comprising:

receiving, at the server, a key frame from the publishing client sharing its screen as a first frame of the stream with the plurality of delta frames following the key frame, wherein the key frame defines a screen of the publishing client in its entirety at the beginning of the screen sharing session with the respective one or more viewing clients.

3. The method of claim 2, wherein each of the plurality of delta frames from the publishing client represent a subset of the screen of the publishing client corresponding to changes at the screen of the publishing client since a preceding delta frame or since a preceding key frame when the delta frame is a first delta frame of the screen sharing session from the publishing client after the key frame.

4. The method of claim 1, wherein each of the plurality of delta frames from the publishing client specifies changes to a screen of the publishing client and includes less than all data required to define the screen of the publishing client in its entirety.

5. The method of claim 1, wherein re-queuing the single aggregated delta frame comprises queuing the single aggregated delta frame in the respective FIFO buffer with a priority according to an original position of the two or more delta frames aggregated.

6. The method of claim 1, further comprising:

establishing a server transmitter for each of the respective one or more viewing clients on 1;

1 basis; and

wherein each of the respective server transmitter is dedicated to transmitting queued delta frames for exactly one of the viewing clients from the corresponding FIFO buffer established for the same viewing client.

7. The method of claim 6, wherein transmitting the single aggregated delta frame to the respective viewing client comprises transmitting the single aggregated delta frame to the respective viewing client from the corresponding FIFO buffer as the delta frames are queued into the FIFO buffer at the greater of:

(a) an unrestricted bandwidth rate at which the delta frames are received into the queue or (b) a restricted bandwidth rate at which each respective viewing client is capable of consuming the delta frames from the server.

8. The method of claim 1, wherein the two or more delta frames become concurrently queued in the FIFO buffers for one of the viewing clients when the viewing client consumes the stream of delta frames from the server slower than the publishing client sends the stream of delta frames to the server.

9. The method of claim 1:

wherein the publishing client sends the stream of delta frames to the server at a rate greater than one of the viewing clients consumes the stream of delta frames from the server; and

wherein aggregating the two or more delta frames into the single aggregated delta frame reduces a quantity of delta frames transmitted to the viewing client and reduces a total payload data transmitted to the viewing client via the stream of delta frames.

10. The method of claim 1:

wherein the server receives the stream of delta frames from the publishing client at a first bandwidth rate;

wherein a first one of the viewing clients receives the stream of delta frames from the server at a second bandwidth rate lesser than the first bandwidth rate via frame aggregation by the server iteratively combine multiple of the delta frames in the stream into aggregated delta frames on behalf of the first one of the viewing clients; and

wherein a second one of the viewing clients receives the stream of delta frames from the server at the first bandwidth rate equal to the rate at which the server receives the stream of delta frames from the publishing client, the second one of the viewing clients to receive the stream of delta frames from the server without frame aggregation on the stream of delta frames by the server.

11. A method performed by a publishing client device, wherein the method comprises:

preparing delta frames for transmission to a server within a stream of delta frames from the publishing client device as part of a screen sharing session with one or more viewing clients communicably interfaced with the server;

wherein the server establishes a FIFO buffer for each of the respective one or more viewing clients on 1;

1 basis to receive the stream of delta frames from the publishing client device;

packaging the delta frames into media object containers by adding the delta frames as payload to the media object containers;

sequencing the media object containers by tagging each of the media object containers with an assigned sequence number;

transmitting, via a client transmitter of the publishing client device, the media object containers having the delta frames therein to the server within the stream of delta frames;

wherein the server receives the media object containers having the delta frames therein in an order different than an original order as transmitted via the client transmitter; and

wherein a sequencer of the server sequences the delta frames into the original order as transmitted via the client transmitter based on the assigned sequence number tagged to each of the media object containers by the publishing client device.

12. The method of claim 11, wherein the server transmits the stream of delta frames to the respective one or more viewing clients from the FIFO buffer established for each of the respective one or more viewing clients.

13. The method of claim 11, further comprising:

placing the media object containers into an output queue of the publishing client device; and

wherein transmitting the media object containers having the delta frames therein to the server within the stream of delta frames comprises transmitting the media object containers to the server from the output queue of the publishing client device.

14. The method of claim 13, further comprising:

periodically checking the output queue of the publishing client device to determine if there are multiple delta frames in the output queue;

aggregating the multiple delta frames into a single aggregated delta frame; and

substituting the multiple delta frames with the single aggregated delta frame within the output queue of the publishing client device.

15. The method of claim 11, wherein the server is operated by a host organization providing on-demand cloud computing services to subscribers of the host organization.

16. The method of claim 15:

wherein the server is one of a plurality of slipstream media servers operated by the host organization;

wherein the host organization facilitates the exchange of information between the publishing client device and the one or more viewing clients as part of a cloud service; and

wherein the cloud service provided by the host organization negates the need for the subscribers of the host organization to install and maintain server-side software locally.

17. The method of claim 11:

wherein sequencing the media object containers by tagging each of the media object containers with the assigned sequence number provides lossless sequencing for lossless transmission of the stream of delta frames from the publishing client device via one or more TCP/IP transmissions.

18. The method of claim 11:

wherein the server monitors each of the respective FIFO buffers for each of the respective one or more viewing clients to determine if two or more delta frames are concurrently queued in any single one of the respective FIFO buffers;

wherein the server aggregates the two or more delta frames into a single aggregated delta frame;

wherein the server re-queues the single aggregated delta frame; and

wherein the server transmits the single aggregated delta frame to the respective one or more viewing clients.

19. The method of claim 11:

wherein the server triggers a re-transmission request to the publishing client device for any of the delta frames delayed for a period of time exceeding a threshold.

20. Non-transitory computer readable storage media having instructions stored thereupon that, when executed by a processor of a publishing client device, the instructions cause the publishing client device to perform operations including:

preparing delta frames for transmission to a server within a stream of delta frames from the publishing client device as part of a screen sharing session with one or more viewing clients communicably interfaced with the server;

wherein the server establishes a FIFO buffer for each of the respective one or more viewing clients on 1;

1 basis to receive the stream of delta frames from the publishing client device;

packaging the delta frames into media object containers by adding the delta frames as payload to the media object containers;

sequencing the media object containers by tagging each of the media object containers with an assigned sequence number;

transmitting, via a client transmitter of the publishing client device, the media object containers having the delta frames therein to the server within the stream of delta frames;

wherein the server receives the media object containers having the delta frames therein in an order different than an original order as transmitted via the client transmitter; and

wherein a sequencer of the server sequences the delta frames into the original order as transmitted via the client transmitter based on the assigned sequence number tagged to each of the media object containers by the publishing client device.

21. The non-transitory computer readable storage media of claim 20, wherein the server transmits the stream of delta frames to the respective one or more viewing clients from the FIFO buffer established for each of the respective one or more viewing clients.