Systems, methods, and apparatuses for accepting late joiners with screen sharing

• US 9,665,331 B2
• Filed: 03/15/2013
• Issued: 05/30/2017
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Spencer, Barry L.
• Primary Examiner(s): Meky, Moustafa M
• Assistant Examiner(s): Islam, SM Z
• Application Number:
  US13/840,282
• Publication Number:
  US 20130346500A1
• Time in Patent Office: 1,537 Days
• Field of Search: 709203
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

In accordance with disclosed embodiments, there are provided methods, systems, and apparatuses for accepting late joiners with screen sharing including, for example, means for receiving, at a server, a key frame from a publishing client sharing its screen, the key frame defining the screen of the publishing client in its entirety at the beginning of a screen sharing session with one or more viewing clients; transmitting the key frame to the one or more viewing clients; iteratively processing each of a plurality of delta frames from the publishing client specifying changes to the screen of the publishing client, wherein the iterative processing includes: (i) receiving each delta frame, (ii) updating an aggregated current key frame with the delta frame received, and (iii) sending the delta frame to the one or more viewing clients. Such means further include: accepting a late joiner viewing client for the screen sharing session; sending the aggregated current key frame to the late joiner viewing client; and sending subsequently received delta frames to the one or more viewing clients and to the late joiner viewing client. Other related embodiments are disclosed.

Claims

1. A method comprising:

receiving, at a server, a key frame from a publishing client sharing its screen, the key frame defining the screen of the publishing client in its entirety at the beginning of a screen sharing session with one or more viewing clients;

transmitting, with the server, the key frame to the one or more viewing clients;

iteratively processing, with the server, each of a plurality of delta frames, wherein the delta frames specify changes to the screen of the publishing client since the key frame or since a previous delta frame and the iterative processing comprises;

(i) maintaining a queue for each of the one or more viewing clients,

(ii) receiving each delta frame from the publishing client,

(iii) storing either a received delta frame or an aggregated delta frame in each queue,

(iv) maintaining an aggregated current key frame that corresponds to the key frame updated with each delta frame received, and

(v) sending the delta frames from the queues to each of the one or more viewing clients;

accepting, with the server, a late joiner viewing client for the screen sharing session;

sending, with the server, the aggregated current key frame to the late joiner viewing client; and

sending, with the server, subsequently received delta frames to the one or more viewing clients and to the late joiner viewing client.

2. The method of claim 1, further comprising:

establishing a communications interface with the publishing client prior to receiving the key frame from the publishing client; and

establishing a communications interface with the one or more viewing clients to share the screen from the publishing client prior to receiving the key frame from the publishing client.

3. The method of claim 2, further comprising:

transmitting the key frame to the one or more viewing clients prior to accepting the late joiner viewing client for the screen sharing session; and

transmitting one or more of the plurality of delta frames from the publishing client to the one or more viewing clients prior to accepting the late joiner viewing client for the screen sharing session.

4. The method of claim 1, further comprising:

storing the key frame from the publishing client sharing its screen at the server as a current key frame for time₀ corresponding to the start of the screen sharing session;

receiving a first of the plurality of delta frames from the publishing client at time₁ ; and

aggregating the first of the plurality of delta frames into the current key frame for time₀ to result in an aggregated current key frame for time₁.

5. The method of claim 4, wherein accepting a late joiner viewing client for the screen sharing session and sending the aggregated current key frame to the late joiner viewing client, comprises:

accepting the late joiner viewing client after time₁;

sending the late joiner viewing client the aggregated current key frame for time₁, wherein the aggregated current key frame for time₁ defines a current state of the screen of the publishing client in its entirety at time time₁;

receiving a second of the plurality of delta frames from the publishing client at time₂;

aggregating the second of the plurality of delta frames into the aggregated current key frame for time₁ to result in an aggregated current key frame for time₂;

storing the aggregated current key frame for time₂; and

sending the second of the plurality of delta frames from the publishing client at time₂ to the one or more viewing clients and to the late joiner viewing client.

6. The method of claim 5, further comprising:

receiving further ones of the plurality of delta frames from the publishing client at time₃ through time_n;

aggregating each of the further ones of the plurality of delta frames into the aggregated current key frame for time₃ through time_n to result in an aggregated current key frame for time_n;

storing the aggregated current key frame for time_n; and

sending the further ones of the plurality of delta frames from the publishing client at time₃ through time_n to the one or more viewing clients and to the late joiner viewing client.

7. The method of claim 1, wherein each of the plurality of delta frames from the publishing client represent a subset of the screen of the publishing client corresponding to changes at the screen of the publishing client since a preceding delta frame or since the preceding key frame when the delta frame is the first delta frame of the screen sharing session from the publishing client after the key frame.

8. The method of claim 1, wherein accepting a late joiner viewing client for the screen sharing session comprises:

accepting the late joiner viewing client when the late joiner viewing client has missed the key frame at a beginning of the screen sharing session and has further missed one or more delta frames subsequent to the key frame.

9. The method of claim 1, wherein the server is hosted within a host organization operating remote from the publishing client, remote from the one or more viewing clients, and remote from the late joiner viewing client.

10. The method of claim 9, wherein the host organization provides screen sharing services as a cloud service over a public Internet to the publishing client, the one or more viewing clients, and the late joiner viewing client.

11. The method of claim 1, further comprising:

receiving the plurality of delta frames from the publishing client at a receive interface of the server;

sequencing the plurality of delta frames from the publishing client according to a frame identifier of each of the plurality of delta frames; and

transmitting the plurality of delta frames to the one or more viewing clients in sequence.

12. The method of claim 1, further comprising:

queuing each of the plurality of delta frames in an outgoing FIFO buffer prior to transmitting the one or more viewing clients;

iteratively checking the outgoing FIFO buffer to determine if two or more of the plurality of delta frames remain concurrently queued for transmitting;

aggregating the two or more delta frames into a single aggregated delta frame;

queuing the aggregated delta frame in the outgoing FIFO buffer with a priority according to its original position in the outgoing FIFO buffer prior to aggregation; and

transmitting the aggregated delta frame to the one or more viewing clients.

13. The method of claim 1, further comprising:

establishing a dedicated server transmitter, frame aggregator, frame sequencer, and FIFO buffer for each of the one or more viewing clients and the late joiner viewing client on a 1;

1 basis;

sequencing and buffering each of the plurality of delta frames into the respective FIFO buffer for each of the one or more viewing clients and the late joiner viewing client;

monitoring each of the respective FIFO buffers for each of the one or more viewing clients and the late joiner viewing client to determine if two or more delta frames are concurrently queued at any given time;

aggregating the two or more delta frames into a single aggregated delta frame;

queuing the aggregated delta frame in the respective FIFO buffer with a priority according to its original position; and

transmitting the aggregated delta frame to the respective viewing client or late joiner viewing client from the corresponding FIFO buffer.

14. A method at a host organization, the method comprising:

establishing a communications interface with a publishing client, the publishing client to share a screen with one or more viewing clients via the host organization;

establishing a communications interface with the one or more viewing clients to share the screen from the publishing client;

receiving a key frame from the publishing client, the key frame defining the screen of the publishing client in its entirety at time₀;

transmitting the key frame to the one or more viewing clients;

receiving a plurality of delta frames from the publishing client, the plurality of delta frames defining a subset of the screen of the publishing client at time₁ through time_n;

generating an aggregated current key frame at the host organization by updating the key frame with changes specified by the delta frames;

sending the plurality of delta frames to the one or more viewing clients to update the screen shared by the publishing client;

receiving a new communications interface from a late joiner viewing client, the late joiner viewing client having missed the key frame and one or more of the delta frames;

sending the aggregated current key frame to the late joiner viewing client defining the screen of the publishing client in its entirety at time_n according to the key frame and all subsequent delta frames;

receiving delta frames from the publishing client after time_n;

aggregating the delta frames to generate an aggregated delta frame; and

sending the aggregated delta frame after time_n to the one or more viewing clients and to the late joiner viewing client.

15. Non-transitory computer readable storage media having instructions stored thereon that, when executed by a processor in a server, the instructions cause the server to perform operations comprising:

receiving, at a server, a key frame from a publishing client sharing its screen, the key frame defining the screen of the publishing client in its entirety at the beginning of a screen sharing session with one or more viewing clients;

transmitting, with the server, the key frame to the one or more viewing clients;

iteratively processing, with the server, each of a plurality of delta frames, wherein the delta frames specify changes to the screen of the publishing client since the key frame or since a previous delta frame and the iterative processing comprises;

(i) maintaining a queue for each of the one or more viewing clients,

(ii) receiving each delta frame from the publishing client,

(iii) storing either a received delta frame or an aggregated delta frame in each queue,

(iv) maintaining an aggregated current key frame that corresponds to the key frame updated with each delta frame received, and

(v) sending the delta frames from the queues to each of the one or more viewing clients;

accepting, with the server, a late joiner viewing client for the screen sharing session;

sending, with the server, the aggregated current key frame to the late joiner viewing client; and

sending, with the server, subsequently received delta frames to the one or more viewing clients and to the late joiner viewing client.

16. The non-transitory computer readable storage media of claim 15, wherein the operations further comprise:

storing the key frame from the publishing client sharing its screen at the server as a current key frame for time₀ corresponding to the start of the screen sharing session;

receiving a first of the plurality of delta frames from the publishing client at time₁; and

aggregating the first of the plurality of delta frames into the current key frame for time₀ to result in an aggregated current key frame for time₁.

17. The non-transitory computer readable storage media of claim 15, wherein accepting a late joiner viewing client for the screen sharing session comprises:

accepting the late joiner viewing client when the late joiner viewing client has missed the key frame at a beginning of the screen sharing session and has further missed one or more delta frames subsequent to the key frame.

18. A server comprising:

a processor to execute a screen sharing service on behalf of a plurality of clients;

a receive interface of the server to receive a key frame from a publishing client sharing its screen with one or more viewing clients via the screen sharing service, the key frame defining the screen of the publishing client in its entirety at the beginning of a screen sharing session;

a transmitter to transmit the key frame to the one or more viewing clients;

the screen sharing service to iteratively process each of a plurality of delta frames, wherein the delta frames specify changes to the screen of the publishing client since the key frame or since a previous delta frame and the iterative processing comprises;

(i) maintaining a queue for each of the one or more viewing clients,

(ii) receiving each delta frame from the publishing client,

(iii) storing either a received delta frame or an aggregated delta frame in each queue,

(iv) maintaining an aggregated current key frame that corresponds to the key frame updated with each delta frame received, and

(v) sending the delta frames from the queues to each of the one or more viewing clients;

wherein the screen sharing service is to accept a late joiner viewing client for the screen sharing session;

the transmitter to send the aggregated current key frame to the late joiner viewing client; and

the transmitter to subsequently send received delta frames to the one or more viewing clients and to the late joiner viewing client.

19. The server of claim 18, further comprising:

a server transmitter, a frame aggregator, a frame sequencer, and a FIFO buffer dedicated to each of the one or more viewing clients and the late joiner viewing client on a 1;

1 basis;

sequencing via the sequencer and buffering via the FIFO buffer, each of the plurality of delta frames into the respective FIFO buffer for each of the one or more viewing clients and the late joiner viewing client;

the screen sharing service to monitor the respective FIFO buffer for each of the one or more viewing clients and the late joiner viewing client to determine if two or more delta frames are concurrently queued at any given time;

the frame aggregator to aggregate the two or more delta frames into a single aggregated delta frame;

the FIFO buffer to queue the aggregated delta frame in the respective FIFO buffer with a priority according to its original position; and

the screen sharing service to transmit the aggregated delta frame to the respective viewing client or late joiner viewing client via the respective transmitter from the corresponding FIFO buffer.

20. The server of claim 18, wherein the screen sharing service to accept the late joiner viewing client comprises:

accepting the late joiner viewing client when the late joiner viewing client has missed the key frame at a beginning of the screen sharing session and has further missed one or more delta frames subsequent to the key frame.

21. The server of claim 18, wherein the server is hosted within a host organization operating remote from the publishing client, remote from the one or more viewing clients, and remote from the late joiner viewing client.

22. The server of claim 21, wherein the host organization provides the screen sharing service as a cloud service over a public Internet to the publishing client, the one or more viewing clients, and the late joiner viewing client and distributes a video stream of the screen sharing session on behalf of the publishing client.