Intelligent augmented audio conference calling using headphones

• US 10,491,643 B2
• Filed: 06/13/2017
• Issued: 11/26/2019
• Est. Priority Date: 06/13/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Brown, Suzanne C., Lorho, Gaetan R., Sheaffer, Jonathan D.
• Primary Examiner(s): Zhang, Zhensheng
• Application Number:
  US15/621,890
• Publication Number:
  US 20180359294A1
• Time in Patent Office: 896 Days
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04M 3/42059 Making use of the calling party identifier
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/306 For headphones

Abstract

In one aspect herein, a pre-processor receives audio signals for a conference call from individual callers, each of the audio signals associated with corresponding metadata, analyzes the metadata, and associates each of the audio signals with a spatial position in a virtual representation of the conference call based on the analyzation of the metadata. A spatial arrangement processor generates a binaural room impulse response associated with the spatial position of each of the audio signals to filter the received audio signals to account for the spatial position associated with each of the audio signals and to account for the effect of the virtual representation of the conference call. A head-tracking controller tracks an orientation of a listener'"'"'s head using a headset. A binaural renderer produces multi-channel audio data for playback on the headset according to the orientation of the listener'"'"'s head and the binaural room impulse response associated with the spatial position of each of the audio signals.

Claims

1. An audio conference calling system including a headset, the audio conference calling system comprising:

a pre-processor for receiving audio signals for a conference call from individual callers, each of the audio signals associated with corresponding metadata, for analyzing the metadata to determine an importance of the individual callers, and for associating each of the audio signals with a spatial position in a virtual representation of the conference call based on the importance of the individual callers, wherein one or more of the audio signals are automatically assigned to spatially dominant positions in the virtual representation based on the importance of the individual callers;

a spatial arrangement processor for generating a binaural room impulse response associated with the spatial position of each of the audio signals and for using the binaural room impulse response to filter the received audio signals to account for the spatial position associated with each of the audio signals and to account for the effect of the virtual representation of the conference call;

a head-tracking controller for tracking an orientation of a listener'"'"'s head, wherein the listener is using the headset; and

a binaural renderer for producing multi-channel audio data for playback on the headset according to the orientation of the listener'"'"'s head and the binaural room impulse response associated with the spatial position of each of the audio signals.

2. The audio conference calling system of claim 1 wherein the pre-processor associates each of the audio signals with a group based on the corresponding metadata, wherein the group of audio signals is associated with a similar type of spatial position in the virtual representation of the conference call.

3. The audio conference calling system of claim 2 wherein the metadata includes a group affiliation and the grouping is based on the group affiliation.

4. The audio conference calling system of claim 1 wherein the metadata includes one or more of the following:

geographic location, time zone, affiliation, signal quality, average volume, frequency of speech.

5. The audio conference calling system of claim 1 wherein the spatial position includes azimuth, elevation and distance.

6. The audio conference calling system of claim 1 wherein the pre-processor associates each of the audio signals with the spatial position using machine learning based on a previous conference call or psycho acoustic knowledge, to determine what individual callers are calling, to determine a preset spatial position to apply to each individual caller, or to determine a spatial position preference for each individual caller.

7. The audio conference calling system of claim 1 wherein analyzing the metadata includes employing a dimensionality reduction algorithm to the metadata to filter metadata to be used to associate each of the audio signals with the spatial position in the virtual representation of the conference call.

8. The audio conference calling system of claim 1 wherein the spatial arrangement processor generates the binaural room impulse response using a dedicated simulation framework.

9. The audio conference calling system of claim 1 wherein the spatial arrangement processor generates environmental cues and simulates a room of the listener to determine a direction of data to be encoded for each individual caller, for integration of the binaural renderer and the head-tracking controller.

10. The audio conference calling system of claim 1 wherein in a case that the spatial arrangement processor identifies at least two clusters of individual callers, the spatial arrangement processor identifies one of the at least two identified clusters of individual callers as an important cluster based on the importance of one or more of the individual callers in the important cluster, and wherein the spatial positioning of the audio signals from the one or more individual callers in the important cluster is based on the importance of the one or more individual callers in the important cluster.

11. The audio conference calling system of claim 1 wherein in a case where one of the individual callers is identified as leading the conversation, the audio signal from the identified individual caller is spatially positioned in front of the listener in the virtual representation.

12. A method for an audio conference calling system including a headset, the method comprising:

receiving, by a pre-processor, audio signals for a conference call from individual callers, each of the audio signals associated with corresponding metadata, analyzing the metadata to determine an importance of the individual callers, and associating each of the audio signals with a spatial position in a virtual representation of the conference call based on the importance of the individual callers, wherein one or more of the audio signals are automatically assigned to spatially dominant positions in the virtual representation based on the importance of the individual callers;

generating, by a spatial arrangement processor, a binaural room impulse response associated with the spatial position of each of the audio signals and using the binaural room impulse response to filter the received audio signals to account for the spatial position associated with each of the audio signals and to account for the effect of the virtual representation of the conference call;

tracking, by a head-tracking controller, an orientation of a listener'"'"'s head, wherein the listener is using the headset; and

producing, by a binaural renderer, multi-channel audio data for playback on the headset according to the orientation of the listener'"'"'s head and the binaural room impulse response associated with the spatial position of each of the audio signals.

13. The method of claim 12 wherein the pre-processor associates each of the audio signals with a group based on the corresponding metadata, wherein the group of audio signals is associated with a similar type of spatial position in the virtual representation of the conference call.

14. The method of claim 13 wherein the metadata includes a group affiliation and the grouping is based on the group affiliation.

15. The method of claim 12 wherein the metadata includes one or more of the following:

geographic location, time zone, signal quality, average volume, frequency of speech.

16. The method of claim 12 wherein the spatial position includes azimuth, elevation and distance.

17. The method of claim 12 wherein the pre-processor associates each of the audio signals with the spatial position using machine learning based on a previous conference call or psycho acoustic knowledge, to determine what individual callers are calling, to determine a preset spatial position to apply to each individual caller, or to determine a spatial position preference for each individual caller.

18. The method of claim 12 wherein analyzing the metadata includes employing a dimensionality reduction algorithm to the metadata to filter metadata to be used to associate each of the audio signals with the spatial position in the virtual representation of the conference call.

19. The method of claim 12 wherein the spatial arrangement processor generates the binaural room impulse response using a dedicated simulation framework.

20. The method of claim 12 wherein the spatial arrangement processor generates environmental cues and simulates a room of the listener to determine a direction of data to be encoded for each individual caller, for integration of the binaural renderer and the head-tracking controller.

21. The method of claim 12 wherein in a case that the spatial arrangement processor identifies at least two clusters of individual callers, the spatial arrangement processor identifies one of the at least two identified clusters of individual callers as an important cluster based on the importance of one or more of the individual callers in the important cluster, and wherein the spatial positioning of the audio signals from the one or more individual callers in the important cluster is based on the importance of the one or more individual callers in the important cluster.

22. The method of claim 12 wherein in a case where one of the individual callers is identified as leading the conversation, the audio signal from the identified individual caller is spatially positioned in front of the listener in the virtual representation.

23. A non-transitory computer-readable storage medium storing executable program instructions which when executed by a data processing system cause the data processing system to perform a method for controlling an audio conference calling system including a headset, the method comprising:

receiving, by a pre-processor, audio signals for a conference call from individual callers, each of the audio signals associated with corresponding metadata, analyzing the metadata to determine an importance of the individual callers, and associating each of the audio signals with a spatial position in a virtual representation of the conference call based on the importance of the individual callers, wherein one or more of the audio signals are automatically assigned to spatially dominant positions in the virtual representation based on the importance of the individual callers;

generating, by a spatial arrangement processor, a binaural room impulse response associated with the spatial position of each of the audio signals and using the binaural room impulse response to filter the received audio signals to account for the spatial position associated with each of the audio signals and to account for the effect of the virtual representation of the conference call;

tracking, by a head-tracking controller, an orientation of a listener'"'"'s head, wherein the listener is using the headset; and

producing, by a binaural renderer, multi-channel audio data for playback on the headset according to the orientation of the listener'"'"'s head and the binaural room impulse response associated with the spatial position of each of the audio signals.