Directional sound modification

• US 10,575,117 B2
• Filed: 06/12/2018
• Issued: 02/25/2020
• Est. Priority Date: 12/08/2014
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Di Censo, Davide, Marti, Stefan, Verbeke, Joseph
• Primary Examiner(s): Bernardi, Brenda C
• Application Number:
  US16/006,783
• Publication Number:
  US 20180302738A1
• Time in Patent Office: 623 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04R 1/1041 Mechanical or electronic switches, or control elements switches in general H01H
• CPC Class Codes: H04R 1/1083 Reduction of ambient noise active noise reduction per se G10K11/175; protective devices for the ear, e.g. providing acoustic protection A61F11/06
• CPC Class Codes: H04R 2203/12 Beamforming aspects for stereophonic sound reproduction with loudspeaker arrays
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2499/13 Acoustic transducers and sound field adaptation in vehicles
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 3/12 for distributing signals to two or more loudspeakers specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 3/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/306 For headphones

Abstract

Various embodiments set forth techniques for directional sound modification. In one aspect, a system includes a plurality of audio sensors configured to acquire sound from an environment and at least one processor coupled to the plurality of audio sensors. The at least one processor is configured to determine a direction within the environment, generate an audio signal based on sound acquired from the direction within the environment, and transmit the audio signal to at least one audio output device to generate audio output. The audio output combines with sound from the direction within the environment to produce a modified sound.

Claims

1. A system, comprising:

a plurality of audio sensors configured to acquire sound from an environment; and

at least one processor coupled to the plurality of audio sensors and configured to;

determine a direction within the environment based on at least one of a first sound type included in the environment, a location of the system within the environment, and visual data acquired from the environment;

generate an audio signal based on sound acquired from the direction within the environment; and

transmit the audio signal to at least one audio output device to generate audio output, wherein the audio output combines with the sound from the direction within the environment to produce a modified sound.

2. The system of claim 1, wherein the at least one processor is further configured to identify, based on the sound acquired from the environment, the first sound type included in the environment.

3. The system of claim 2, wherein the at least one processor is configured to determine the direction within the environment by identifying the direction within the environment from which the first sound type was acquired by the plurality of audio sensors.

4. The system of claim 2, wherein the at least one processor is configured to determine the direction within the environment by identifying at least one audio sensor included in the plurality of audio sensors that acquired the first sound type, wherein the direction within the environment corresponds to a direction from which the at least one audio sensor acquired sound.

5. The system of claim 2, wherein the at least one processor is configured to determine the direction within the environment by:

identifying a plurality of sound types that includes the first sound type;

determining that the plurality of sound types correspond to a sound scene included in a blacklist; and

identifying the direction within the environment from which the plurality of sound types was acquired by the plurality of audio sensors.

6. The system of claim 1, wherein the at least one processor is configured to determine the direction within the environment by:

obtaining, via at least one location device, a first location of the system;

identifying the first sound type included in the environment based on the first location of the system;

determining that the first sound type is included in a blacklist; and

determining the direction within the environment based on the location of the system and a location of the first sound type.

7. The system of claim 6, wherein the at least one processor is further configured to:

obtain, via the at least one location device, a second location of the system, wherein the second location of the system is different than the first location of the system;

identify at least one second sound type included in the environment based on the second location of the system;

determine that the at least one second sound type is included in the blacklist;

determine a second direction within the environment based on the second location of the system and a location of the at least one second sound type;

generate a second audio signal based on sound acquired from the second direction within the environment; and

transmit the second audio signal to the at least one audio output device to generate second audio output, wherein the second audio output combines with the sound from the second direction within the environment to produce a second modified sound.

8. The system of claim 1, wherein the at least one processor is configured to determine the direction within the environment by:

acquiring, via at least one image sensor, visual data of the environment;

identifying the first sound type based on the visual data;

determining that the at least one sound type is included in a blacklist; and

determining the direction within the environment based on a location in the visual data associated with the at least one sound type.

9. A method for directional sound modification, comprising:

acquiring sound from an environment via a plurality of audio sensors;

determining a direction within the environment based on at least one of a first sound type included in the environment, a location of the system within the environment, and visual data acquired from the environment;

generating an audio signal based on sound acquired from the direction within the environment; and

transmitting the audio signal to at least one audio output device to generate audio output, wherein the audio output combines with the sound from the direction within the environment to produce a modified sound.

10. The method of claim 9, further comprising identifying the first sound type included in the environment.

11. The method of claim 10, wherein identifying the first sound type comprises:

obtaining, via at least one image sensor, visual data of the environment; and

identifying, in the visual data, at least one object that corresponds to the first sound type.

12. The method of claim 10, wherein identifying the first sound type comprises:

obtaining location data indicating a location associated with the plurality of audio sensors; and

identifying the first sound type based on the location associated with the plurality of audio sensors.

13. The method of claim 12, further comprising:

obtaining second location data indicating a second location associated with the plurality of audio sensors;

determining a second direction within the environment based on the second location associated with the plurality of audio sensors and a location that corresponds to the first sound type;

generating a second audio signal based on sound acquired from the second direction within the environment; and

transmitting the second audio signal to the at least one audio output device to generate second audio output, wherein the second audio output combines with the sound from the second direction within the environment to produce a second modified sound.

14. The method of claim 10, wherein identifying the first sound type comprises comparing the sound acquired from the environment to a database of sounds.

15. One or more non-transitory computer readable storage media storing instructions, that, when executed by at least one processor, cause the at least one processor to perform the steps of:

acquiring, via at least one audio sensor included in a plurality of audio sensors, a first sound from an environment;

identifying a first sound type included in the environment based on the first sound;

determining a direction within the environment based on at least one of the first sound type, a location of the system within the environment, and visual data acquired from the environment, wherein the first sound is acquired from the direction within the environment;

generating an audio signal based on the first sound; and

transmitting the audio signal to at least one audio output device to generate audio output, wherein the audio output combines with the first sound to produce a modified sound.

16. The one or more non-transitory computer readable storage media of claim 15, wherein determining the direction within the environment comprises determining that the first sound was acquired via the at least one audio sensor, wherein the at least on audio sensor is oriented to acquire sound from the direction within the environment.

17. The one or more non-transitory computer readable storage media of claim 15, wherein identifying the first sound type comprises applying at least one machine learning algorithm to the first sound.

18. The one or more non-transitory computer readable storage media of claim 15, wherein identifying the first sound type comprises:

obtaining visual data of the environment; and

identifying the first sound type based on the first sound and the visual data.

19. The one or more non-transitory computer readable storage media of claim 15, wherein identifying the first sound type comprises:

obtaining location data indicating a location associated with the plurality of audio sensors; and

identifying the first sound type based on the first sound and the location associated with the plurality of audio sensors.