Processing audio data to compensate for partial hearing loss or an adverse hearing environment

• US 10,136,240 B2
• Filed: 04/19/2016
• Issued: 11/20/2018
• Est. Priority Date: 04/20/2015
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: De Burgh, Mark David, Yap, Tet Fei
• Primary Examiner(s): Tran, Thang
• Application Number:
  US15/568,451
• Publication Number:
  US 20180115850A1
• Time in Patent Office: 945 Days
• CPC Class Codes: H04R 2227/001 Adaptation of signal processing in PA systems in dependence of presence of noise
• CPC Class Codes: H04R 2227/009 Signal processing in [PA] systems to enhance the speech intelligibility
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04R 3/12 for distributing signals to two or more loudspeakers specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

Methods a provided for improving an audio scene for people suffering from hearing loss or for adverse hearing environments. Audio objects may be prioritized. In some implementations, audio objects that correspond to dialog may be assigned to a highest priority level. Other implementations may involve assigning the highest priority to other types of audio objects, such as audio objects that correspond to events. During a process of dynamic range compression, higher-priority objects may be boosted more, or cut less, than lower-priority objects. Some lower-priority audio objects may fall below the threshold of human hearing, in which case the audio objects may be dropped and not rendered.

Claims

1. A method, comprising:

receiving audio data comprising a plurality of audio objects, the audio objects including audio signals and associated audio object metadata, the audio object metadata including audio object position metadata;

receiving reproduction environment data comprising an indication of a number of reproduction speakers in a reproduction environment;

determining at least one audio object type from among a list of audio object types that includes dialogue;

making an audio object prioritization based, at least in part, on the audio object type, wherein making the audio object prioritization involves assigning a highest priority to audio objects that correspond to the dialogue;

adjusting audio object levels according to the audio object prioritization; and

rendering the audio objects into a plurality of speaker feed signals based, at least in part, on the audio object position metadata, wherein each speaker feed signal corresponds to at least one of the reproduction speakers within the reproduction environment,

wherein rendering involves rendering the audio objects to locations in a virtual acoustic space and increasing a distance between at least some audio objects in the virtual acoustic space.

2. The method of claim 1, further comprising receiving hearing environment data comprising at least one factor selected from a group of factors consisting of:

a model of hearing loss;

a deficiency of at least one reproduction speaker; and

current environmental noise, wherein adjusting the audio object levels is based, at least in part, on the hearing environment data.

3. The method of claim 1, wherein the virtual acoustic space includes a front area and a back area and wherein the rendering involves increasing a distance between at least some audio objects in the front area of the virtual acoustic space.

4. The method of claim 3, wherein the virtual acoustic space is represented by spherical harmonics, and the method comprises increasing the angular separation between at least some audio objects in the front area of the virtual acoustic space prior to rendering.

5. The method of claim 1, wherein the rendering involves rendering the audio objects according to a plurality of virtual speaker locations within the virtual acoustic space.

6. The method of claim 1, wherein the audio object metadata includes metadata indicating audio object size and wherein making the audio object prioritization involves applying a function that reduces a priority of non-dialogue audio objects according to increases in audio object size.

7. The method of claim 1, further comprising:

determining that an audio object has audio signals that include a directional component and a diffuse component; and

reducing a level of the diffuse component.

8. A method, comprising:

receiving audio data comprising a plurality of audio objects, the audio objects including audio signals and associated audio object metadata;

extracting one or more features from the audio data;

determining an audio object type based, at least in part, on features extracted from the audio signals, wherein the audio object type is selected from a list of audio object types that includes dialogue;

making an audio object prioritization based, at least in part, on the audio object type, wherein the audio object prioritization determines, at least in part, a gain to be applied during a process of rendering the audio objects into speaker feed signals, the process of rendering involving rendering the audio objects to locations in a virtual acoustic space, and wherein making the audio object prioritization involves assigning a highest priority to audio objects that correspond to the dialogue;

adding audio object prioritization metadata, based on the audio object prioritization, to the audio object metadata; and

increasing a distance between at least some audio object in the virtual acoustic space.

9. The method of claim 8, wherein the one or more features include at least one feature from a list of features consisting of:

spectral flux;

loudness;

audio object size;

entropy-related features;

harmonicity features;

spectral envelope features;

phase features; and

temporal features.

10. The method of claim 8, further comprising:

determining a confidence score regarding each audio object type determination; and

applying a weight to each confidence score to produce a weighted confidence score, the weight corresponding to the audio object type determination, wherein making an audio object prioritization is based, at least in part, on the weighted confidence score.

11. The method of claim 8, further comprising:

receiving hearing environment data comprising a model of hearing loss;

adjusting audio object levels according to the audio object prioritization and the hearing environment data; and

rendering the audio objects into a plurality of speaker feed signals based, at least in part, on the audio object position metadata, wherein each speaker feed signal corresponds to at least one of the reproduction speakers within the reproduction environment.

12. The method of claim 8, wherein the audio object metadata includes audio object size metadata and wherein the audio object position metadata indicates locations in a virtual acoustic space, further comprising:

receiving hearing environment data comprising a model of hearing loss;

receiving indications of a plurality of virtual speaker locations within the virtual acoustic space;

adjusting audio object levels according to the audio object prioritization and the hearing environment data; and

rendering the audio objects to the plurality of virtual speaker locations within the virtual acoustic space based, at least in part, on the audio object position metadata and the audio object size metadata.

13. An apparatus, comprising:

an interface system capable of receiving audio data comprising a plurality of audio objects, the audio objects including audio signals and associated audio object metadata, the audio object metadata including at least audio object position metadata; and

a control system configured for;

receiving reproduction environment data comprising an indication of a number of reproduction speakers in a reproduction environment;

determining at least one audio object type from among a list of audio object types that includes dialogue;

making an audio object prioritization based, at least in part, on the audio object type, wherein making the audio object prioritization involves assigning a highest priority to audio objects that correspond to the dialogue;

adjusting audio object levels according to the audio object prioritization; and

rendering the audio objects into a plurality of speaker feed signals based, at least in part, on the audio object position metadata, wherein each speaker feed signal corresponds to at least one of the reproduction speakers within the reproduction environment,

wherein rendering involves rendering the audio objects to locations in a virtual acoustic space, and increasing a distance between at least some audio objects in the virtual acoustic space.

14. A non-transitory medium having software stored thereon, the software including instructions for controlling at least one device for:

receiving audio data comprising a plurality of audio objects, the audio objects including audio signals and associated audio object metadata, the audio object metadata including at least audio object position metadata;

receiving reproduction environment data comprising an indication of a number of reproduction speakers in a reproduction environment;

determining at least one audio object type from among a list of audio object types that includes dialogue;

making an audio object prioritization based, at least in part, on the audio object type, wherein making the audio object prioritization involves assigning a highest priority to audio objects that correspond to the dialogue;

adjusting audio object levels according to the audio object prioritization; and

rendering the audio objects into a plurality of speaker feed signals based, at least in part, on the audio object position metadata, wherein each speaker feed signal corresponds to at least one of the reproduction speakers within the reproduction environment,

wherein rendering involves rendering the audio objects to locations in a virtual acoustic space and increasing a distance between at least some audio objects in the virtual acoustic space.

15. An apparatus, comprising:

an interface system capable of receiving audio data comprising a plurality of audio objects, the audio objects including audio signals and associated audio object metadata; and

a control system configured for;

extracting one or more features from the audio data;

determining an audio object type based, at least in part, on features extracted from the audio signals, wherein the audio object type is selected from a list of audio object types that includes dialogue;

making an audio object prioritization based, at least in part, on the audio object type, wherein the audio object prioritization determines, at least in part, a gain to be applied during a process of rendering the audio objects into speaker feed signals, the process of rendering involving rendering the audio objects to locations in a virtual acoustic space, and wherein making the audio object prioritization involves assigning a highest priority to audio objects that correspond to the dialogue;

adding audio object prioritization metadata, based on the audio object prioritization, to the audio object metadata; and

increasing a distance between at least some audio objects in the virtual acoustic space.