In-vehicle gesture interactive spatial audio system

• US 10,126,823 B2
• Filed: 12/31/2014
• Issued: 11/13/2018
• Est. Priority Date: 01/03/2014
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Nahman, Jaime Elliot, Di Censo, Davide, Marti, Stefan
• Primary Examiner(s): Levy, Amy M
• Application Number:
  US14/587,602
• Publication Number:
  US 20150193197A1
• Time in Patent Office: 1,413 Days
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/162 Interface to dedicated audio devices, e.g. audio drivers, interface to CODECs
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04R 2499/13 Acoustic transducers and sound field adaptation in vehicles
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/304 For headphones

Abstract

One embodiment of the present invention sets forth a technique for modifying an audio parameter based on a gesture. The technique includes acquiring sensor data associated with a hand of a user and analyzing the sensor data to determine at least one hand position. The technique further includes determining, based on the at least one hand position, an interaction between a first virtual object that corresponds to an audio event and a second virtual object that corresponds to the hand of the user. The technique further includes, based on the interaction, modifying a spatial audio parameter associated with the audio event to generate a modified audio stream and causing the modified audio stream to be reproduced for output to the user.

Claims

1. A method for modifying an audio parameter based on a gesture, the method comprising:

acquiring sensor data associated with a hand of a user;

analyzing the sensor data to determine at least one hand position;

displaying a first virtual object that comprises a visual representation of an audio event in a first region of a volumetric display, wherein the volumetric display allows the user to interact with virtual objects associated with different audio events and re-position the virtual objects within a virtual three-dimensional space in order to modify positions at which the different audio events are reproduced within a sound space;

displaying a notification associated with a second audio event in the first region, wherein the notification overlaps the first virtual object in the first region;

determining, based on the at least one hand position, an interaction between the first virtual object and a second virtual object that corresponds to the hand of the user;

in response to the interaction,

determining that the interaction is associated with the notification;

determining, based on at least one of a velocity and a duration of the interaction, a first sub-region of the first region of the volumetric display;

displaying the first virtual object within only the first sub-region of the first region;

determining a second sub-region of the first region of the volumetric display that is available to display a third virtual object that comprises a visual representation of the second audio event, wherein the second sub-region is independent of the first sub-region; and

displaying the third virtual object in the second sub-region;

based on the first sub-region, modifying a spatial audio parameter associated with the audio event to generate a modified audio stream; and

causing the modified audio stream to be reproduced for output within an audio region of the sound space corresponding to the first sub-region.

2. The method of claim 1, wherein modifying the spatial audio parameter comprises moving the audio event from a third location within a sound space towards a fourth location within the sound space.

3. The method of claim 1, wherein determining the interaction comprises detecting, based on the coordinates associated with the hand of the user, that the second virtual object is proximate to the first virtual object.

4. The method of claim 1, wherein determining the interaction comprises detecting that the second virtual object is proximate to the first virtual object.

5. The method of claim 4, further comprising causing at least one of auditory feedback and haptic feedback to be generated in response to detecting that the second virtual object is proximate to the first virtual object.

6. The method of claim 2, wherein analyzing the sensor data comprises determining a first hand position at which a grasping gesture was initiated, and determining a second hand position at which the grasping gesture was terminated, wherein the first hand position is associated with the third location within the sound space and the second hand position is associated with the fourth location within the sound space.

7. The method of claim 1, wherein the audio event comprises an ambient audio source that spans a portion of the sound space, and the visual representation spans a portion of the display that corresponds to the portion of the sound space.

8. The method of claim 1, further comprising:

determining, based on the at least one hand position, a second interaction between the first virtual object and the second virtual object;

based on the second interaction, moving the audio event from a first sound space associated with the user to a second sound space associated with a second user to generate a second modified audio stream; and

causing the second modified audio stream to be reproduced within the second sound space.

17. The method of claim 1, further comprising causing a second audio stream to be reproduced for output within a second audio region of the sound space corresponding to the second sub-region.

18. The method of claim 1, wherein the first region comprises all of a display region of the volumetric display.

9. A system for modifying an audio parameter based on a gesture, comprising:

at least one sensor configured to acquire sensor data associated with a hand of a user;

a processor coupled to the at least one sensor and configured to;

analyze the sensor data to determine at least one hand position; and

display a first virtual object that comprises a visual representation of an audio event in a first region of a volumetric display, wherein the volumetric display allows the user to interact with virtual objects associated with different audio events and re-position the virtual objects within a virtual three-dimensional space in order to modify positions at which the different audio events are reproduced within a sound space;

display a notification associated with a second audio event in the first region, wherein the notification overlaps the first virtual object in the first region;

determine, based on the at least one hand position, an interaction between the first virtual object and a second virtual object that corresponds to the hand of the user;

in response to the interaction,

determine that the interaction is associated with the notification;

determine, based on at least one of a velocity and a duration of the interaction, a first sub-region of the first region the volumetric display; and

determine a second sub-region of the first region of the volumetric display that is available to display a third virtual object that comprises a visual representation of the second audio event, wherein the second sub-region is independent of the first sub-region;

based on the sub-region, modify a spatial audio parameter associated with the audio event to generate a modified audio stream; and

cause the modified audio stream to be reproduced for output to the user within an audio region of the sound space corresponding to the sub-region; and

a display coupled to the processor and configured to;

display a visual representation of the first virtual object in the first region; and

in response to the interaction,

display the visual representation within only the sub-region of the first region; and

display the third virtual object in the second sub-region.

10. The system of claim 9, wherein the processor is configured to modify the spatial audio parameter by moving the audio event from a third location within a sound space towards a fourth location within the sound space.

11. The system of claim 9, wherein the sensor data comprises first coordinates associated with the hand of the user in three-dimensional space, and the processor is configured to determine the interaction by determining that the first coordinates associated with the hand of the user are proximate to second coordinates associated with the second virtual object.

12. The system of claim 9, wherein the processor is configured to determine the interaction by determining that the hand of the user is performing a gesture proximate to the three-dimensional image.

13. The system of claim 9, wherein the at least one hand position comprises first hand positions associated with the user, and second hand positions associated with a second user, and the processor is further configured to:

determine, based on the second hand positions, a second interaction between a third virtual object that corresponds to a second audio event and a fourth virtual object that corresponds to a hand of the second user;

based on the second interaction, moving the second audio event from a second sound space associated with the second user to a first sound space associated with the user.

14. The system of claim 9, wherein the processor is further configured to detect, based on the at least one hand position, that the second virtual object is proximate to the first virtual object and, in response, cause at least one of visual feedback, auditory feedback, and haptic feedback to be generated.

15. The system of claim 9, wherein the audio event comprises an ambient audio source, and the processor is configured to modify the spatial audio parameter by modifying a portion of a sound space that is spanned by the audio event.

16. A non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to modify an audio parameter based on a gesture, by performing the steps of:

acquiring sensor data associated with a hand of a user;

analyzing the sensor data to determine at least one hand position;

displaying a first virtual object that comprises a visual representation of an audio event in a first region of a volumetric display, wherein the volumetric display allows the user to interact with virtual objects associated with different audio events and re-position the virtual objects within a virtual three-dimensional space in order to modify positions at which the different audio events are reproduced within a sound space;

displaying a notification associated with a second audio event in the first region, wherein the notification overlaps the first virtual object in the first region;

determining, based on the at least one hand position, an interaction between the first virtual object and a second virtual object that corresponds to the hand of the user;

in response to the interaction,

determining that the interaction is associated with the notification;

determining, based on at least one of a velocity and a duration of the interaction, a first sub-region of the first region of the volumetric display;

displaying the first virtual object within only the first sub-region of the first region;

determining a second sub-region of the first region of the volumetric display that is available to display a third virtual object that comprises a visual representation of the second audio event, wherein the second sub-region is independent of the first sub-region; and

displaying the third virtual object in the second sub-region;

based on the first sub-region, modifying a spatial audio parameter associated with the audio event to generate a modified audio stream; and

causing the modified audio stream to be reproduced for output within an audio region of the sound space corresponding to the first sub-region.