Dynamic locale based aggregation of full duplex media streams

• US 9,641,576 B2
• Filed: 12/23/2014
• Issued: 05/02/2017
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: LaFata, Christopher, Klingbeil, Michael, Marks, Adam, Wiegner, Carlin
• Primary Examiner(s): Patel, Hemant
• Application Number:
  US14/580,866
• Publication Number:
  US 20160014373A1
• Time in Patent Office: 861 Days
• Field of Search: 348 1401- 1416, 370259-271, 370351-357, 709201-207, 709217-248
• CPC Class Codes: H04L 5/14 Two-way operation using the same type of signal, i.e. duplex
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04M 2201/22 Synchronisation circuits
• CPC Class Codes: H04M 2203/5072 Multiple active speakers
• CPC Class Codes: H04M 2242/14 with services dependent on location
• CPC Class Codes: H04M 3/56 Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities video conference systems H04N7/15
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04M 9/082 using echo cancellers echo cancellers per se H04B3/23
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A cloud-based video/audio conferencing system and method performs locale based aggregation of a full duplex media stream to organize multiple connections to a conference call that originate from the same physical location or a shared locale. The cloud-based video/audio conferencing system performs synchronization of the microphone and speaker audio signals of the same-locale connections. In this manner, a conference call may be held with multiple user devices making connections from the same physical location. User experience is enhanced by allowing each user in the same location to use his/her own individual devices to connect to the same conference call.

Claims

1. A method for processing audio signals of a conference call comprising:

detecting, at a conferencing server, a plurality of audio clients connecting to the conference call from a same locale;

selecting an audio client aggregation node at the same locale;

connecting the plurality of audio clients to the audio client aggregation node;

receiving at the audio client aggregation node, microphone audio signals from the connected audio clients, each microphone audio signal containing an audio timestamp based on an audio client timebase;

synchronizing, at the audio client aggregation node, the microphone audio signals using a clock of the audio client aggregation node and the audio timestamp based on the audio client timebase;

receiving, at the audio client aggregation node, speaker audio signals destined for the connected audio clients; and

synchronizing, at the audio client aggregation node, the speaker audio signals.

2. The method of claim 1, wherein in response to detecting a plurality of audio clients connecting to the conference call from a same locale and selecting an audio client aggregation node at the same locale, the method further comprises:

providing to each audio client a list of at least one available audio client aggregation node at the same locale; and

connecting at least one audio client to an audio client aggregation node in the same locale.

3. The method of claim 1, wherein in response to detecting a plurality of audio clients connecting to the conference call from a same locale and selecting an audio client aggregation node at the same locale, the method further comprises:

in response to receiving at the conferencing server a request from a first audio client of the plurality of audio clients to join the conference call, directing the first audio client to connect to the audio client aggregation node at the same locale.

4. The method of claim 1, wherein detecting a plurality of audio clients connecting to the conference call from a same locale comprises:

producing a super audible sound signature from the audio client aggregation node in the same locale; and

in response to detecting in the microphone signal of a first audio client the super audible sound signature, determining that the first audio client is located in the same locale as the audio client aggregation node.

5. The method of claim 1, wherein detecting a plurality of audio clients connecting to the conference call from a same locale comprises:

sampling a background noise signature of each of the plurality of audio clients;

in response to detecting in the microphone signal of a first audio client the same background noise signature as a second audio client, determining that the first audio client and the second audio client are located in the same locale.

6. The method of claim 1, wherein selecting an audio client aggregation node at the same locale comprises:

selecting a first audio client of the plurality of audio clients as the audio client aggregation node, wherein the audio client aggregation node receives audio signals from the first audio client and also receives audio signals from other audio clients connected to the audio client aggregation node in the same locale.

7. The method of claim 1, wherein selecting an audio client aggregation node at the same locale comprises:

selecting a dedicated device in the same locale as the audio client aggregation node.

8. The method of claim 1, wherein selecting an audio client aggregation node at the same locale comprises:

selecting an aggregation node at the conferencing server as the audio client aggregation node.

9. The method of claim 1, wherein synchronizing, at the audio client aggregation node, the microphone audio signals comprises:

assigning a clock of the audio client aggregation node as a common timebase;

determining, at the audio client aggregation node, a clock offset value for each audio client with reference to the common timebase;

translating, for each audio client, the audio timestamp of the microphone audio data packets from the audio client timebase to the common timebase using the clock offset value for the respective audio client; and

synchronizing the microphone audio signals of the plurality of audio clients using the translated audio timestamps.

10. The method of claim 9, further comprising:

comparing the translated audio timestamps of the microphone audio signals of the plurality of audio clients;

selecting an oldest translated audio timestamp of the microphone audio data packets as a reference timestamp;

determining delays for each of the plurality of audio clients based on the translated audio timestamp and the reference timestamp; and

applying the delays to the microphone audio signals of the plurality of audio clients to synchronize the microphone audio signals.

11. The method of claim 9, further comprising:

receiving, at the audio client aggregation node, in the microphone audio data packet a server ping time echo based on the common timebase, the server ping time echo being derived from a server ping time sent to the audio clients in previous speaker audio data packets; and

determining, at the audio client aggregation node, the clock offset value for each audio client with reference to the common timebase using at least the server ping time echo information.

12. The method of claim 1, wherein synchronizing, at the audio client aggregation node, the speaker audio signals comprises:

receiving, at the audio client aggregation node, an audio client playout delay value in the microphone audio data packet associated with each audio client; and

applying the playout delays to the speaker audio signals of the plurality of audio clients to synchronize the speaker audio signals.

13. The method of claim 1, wherein synchronizing the microphone audio signals and the speaker audio signals comprises:

synchronizing, at the audio client aggregation node, the microphone audio signals and the speaker audio signals concurrently.

14. An audio conferencing system for processing audio signals of a conference call, comprising:

a processor configured to;

detect, at a conferencing server, a plurality of audio clients connecting to the conference call from a same locale;

select an audio client aggregation node at the same locale;

connect the plurality of audio clients to the audio client aggregation node;

receive, at the audio client aggregation node, microphone audio signals from the connected audio clients, each microphone audio signal containing an audio timestamp based on an audio client timebase;

synchronize, at the audio client aggregation node, the microphone audio signals using a clock of the audio client aggregation node and the audio timestamp based on the audio client timebase;

receive, at the audio client aggregation node, speaker audio signals destined for the connected audio clients; and

synchronize, at the audio client aggregation node, the speaker audio signals; and

a memory coupled to the processor and configured to provide the processor with instructions.

15. The audio conferencing system of claim 14, wherein synchronizing, at the audio client aggregation node, the microphone audio signals comprises:

assign a clock of the audio client aggregation node as a common timebase;

determine, at the audio client aggregation node, a clock offset value for each audio client with reference to the common timebase;

translate, for each audio client, the audio timestamp of the microphone audio data packets from the audio client timebase to the common timebase using the clock offset value for the respective audio client; and

synchronize microphone audio signals of the plurality of audio clients from the microphone audio data packets using the translated audio timestamps.

16. The audio conferencing system of claim 14, wherein the processor is configured to:

receive an audio playout delay value in the microphone audio data packet associated with each audio client; and

apply the playout delay to the speaker audio signals of the plurality of audio clients to synchronize the speaker audio signals.

17. The audio conferencing system of claim 14, wherein one of at least two audio client nodes at the same locale is selected as the audio client aggregation node.

19. The method of claim 17, further comprising:

selecting one of the at least two audio client nodes at the shared locals as the audio client aggregation node.

20. The method of claim 17, further comprising:

selecting the conferencing server as the audio client aggregation node.

21. The method of claim 17, further comprising:

receiving, at the audio client aggregation node, a mixed audio stream from the audio conference server;

separating, at the audio client aggregation node, the mixed audio stream into speaker data packets for each of the plurality of audio client nodes, wherein the speaker data packets are aligned in the mixed audio stream received from the audio conference server;

synchronizing, at the audio client aggregation node, the speaker data packets for each of the plurality of audio client nodes by de-aligning each speaker data packet specific to the associated audio client node; and

outputting, at the audio client aggregation node, to each audio client node speaker de-aligned data packets from the plurality of audio client nodes except for the given audio client node'"'"'s own audio stream.

18. A method comprising:

detecting, at a conferencing server, a plurality of audio client nodes connecting to a conference call using a variety of connection media, wherein at least two of the audio client nodes are at a shared locale, and wherein each audio client node includes an audio client executing on an individual user device;

connecting the at least two audio client nodes at the shared locale to an audio client aggregation node;

receiving, at the audio client aggregation node, microphone audio signals including microphone data packets from the at least two audio client nodes at the shared locale, each microphone data packet including an audio timestamp based on a client timebase of the associated audio client node;

determining, at the audio client aggregation node, a clock offset value for each audio client with reference to a common timebase of the audio client aggregation node;

translating, for each audio client, the audio timestamp to the common timebase using the clock offset value for the respective audio client;

synchronizing, at the audio client aggregation node, using the corresponding translated audio timestamps microphone audio signals received from the at least two audio client nodes at the shared local connected to the audio client aggregation node;

aggregating, at the audio client aggregation node, the synchronized microphone audio signals into a single audio stream; and

outputting, at the audio client aggregation node, the single audio stream to the conferencing server.