Gesture interactive wearable spatial audio system

• US 10,585,486 B2
• Filed: 12/31/2014
• Issued: 03/10/2020
• Est. Priority Date: 01/03/2014
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Coleman, Tim, Marti, Stefan, Di Censo, Davide
• Primary Examiner(s): Butcher, Brian M
• Application Number:
  US15/107,774
• Publication Number:
  US 20160320847A1
• Time in Patent Office: 1,896 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/162 Interface to dedicated audio devices, e.g. audio drivers, interface to CODECs
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04R 2499/13 Acoustic transducers and sound field adaptation in vehicles
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/304 For headphones

Abstract

One embodiment of the present invention sets forth a technique for modifying an audio parameter based on a gesture. The technique includes acquiring sensor data associated with a hand of a user and analyzing the sensor data to determine at least one hand position. The technique further includes detecting a hand gesture based on the at least one hand position and, in response to the hand gesture, modifying a spatial audio parameter associated with an audio stream to generate a modified audio stream. The technique further includes causing the modified audio stream to be reproduced for output to the user.

Claims

1. A method for modifying an audio parameter based on a gesture, the method comprising:

acquiring, via at least one sensor included in a head-mounted device, sensor data associated with a hand of a user;

analyzing, via a processor, the sensor data to determine at least one hand position;

causing haptic feedback to be generated in response to detecting an intersection between the at least one hand position and a virtual object associated with an audio event included in an audio stream;

detecting a hand gesture based on the at least one hand position;

in response to the hand gesture, modifying, via a processor, a spatial audio parameter associated with the audio event to generate a modified audio stream by moving a first audio event included in the audio stream towards a first side of a sound space and moving a second audio event included in the audio stream towards a second side of the sound space; and

causing the modified audio stream to be reproduced for output to the user.

2. The method of claim 1, wherein analyzing the sensor data comprises determining a first hand position at which the hand gesture was initiated, and determining a second hand position at which the hand gesture was completed.

3. The method of claim 2, wherein modifying the spatial audio parameter comprises moving the audio event from a first location in a sound space associated with the first hand position towards a second location in the sound space associated with the second hand position.

4. The method of claim 3, wherein the hand gesture comprises a grasping gesture.

5. The method of claim 3, wherein modifying the spatial audio parameter further comprises determining, via a physics model, an interaction between a first virtual object that corresponds to the audio event and a second virtual object that corresponds to the hand of the user.

6. The method of claim 1, wherein the sensor data comprises at least one of image data and depth data.

17. The method of claim 1, wherein the head-mounted device comprises at least one of a set of headphones and a pair of eyeglasses.

18. The method of claim 1, wherein the at least one sensor comprises a first sensor and a second sensor, and the first sensor and the second sensor are configured to acquire a stereo image of the hand of the user.

7. A system for modifying an audio parameter based on a gesture, comprising:

a head-mounted device;

one or more sensors included in the head-mounted device and configured to acquire sensor data associated with a hand of a user;

a processor coupled to the one or more sensors and configured to;

analyze the sensor data to determine at least one hand position;

cause haptic feedback to be generated in response to detecting an intersection between the at least one hand position and a virtual object associated with an audio event included in an audio stream;

detect a hand gesture based on the at least one hand position; and

in response to the hand gesture, move a first audio event included in an audio stream from a first location in a sound space towards a second location in the sound space, wherein the second location corresponds to a first side of the sound space, and move a second audio event included in the audio stream towards a second side of the sound space, to generate a modified audio stream; and

speakers coupled to the processor and configured to reproduce the modified audio stream.

8. The system of claim 7, wherein the hand gesture comprises a splitting gesture.

9. The system of claim 7, wherein the processor is configured to analyze the sensor data to determine a first hand position at which the hand gesture was initiated and a second hand position at which the hand gesture was completed, and wherein the first location in the sound space is associated with the first hand position, and the second location in the sound space is associated with the second hand position.

10. The system of claim 9, wherein the hand gesture comprises at least one of a pushing gesture and a swiping gesture.

11. The system of claim 9, wherein the processor is configured to move the first audio event by determining a distance between the first hand position and the second hand position.

12. The system of claim 7, wherein the one or more sensors comprise a first sensor and a second sensor, and wherein the processor is configured to determine the at least one hand position by determining a first position of the first sensor relative to a second position of the second sensor.

13. The system of claim 7, wherein the processor is configured to move the first audio event from the first location in the sound space towards the second location in the sound space based on a head-related transfer function.

14. The system of claim 7, further comprising a feedback device configured to produce haptic feedback in response to detecting an intersection between the at least one hand position and a virtual object associated with the first audio event.

15. The system of claim 7, wherein the processor is further configured to cause the speakers to produce auditory feedback in response to detecting an intersection between the at least one hand position and a virtual object associated with the first audio event.

16. A non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to modify an audio parameter based on a gesture, by performing the steps of:

acquiring, via at least one sensor included in a head-mounted device, sensor data associated with a hand of a user;

analyzing the sensor data to determine a first hand position and a second hand position;

detecting a hand gesture based on the first hand position and the second hand position, wherein the hand gesture comprises a splitting gesture;

in response to the splitting gesture, modifying a spatial audio parameter associated with an audio stream to generate a modified audio stream by moving a first audio event included in the audio stream towards a first side of a sound space and moving a second audio event included in the audio stream towards a second side of the sound space; and

causing the modified audio stream to be reproduced for output to the user.

19. A method for modifying an audio parameter based on a gesture, the method comprising:

acquiring, via at least one sensor included in a head-mounted device, sensor data associated with a hand of a user;

analyzing, via a processor, the sensor data to determine at least one hand position by;

determining a first hand position at which a hand gesture was initiated, and

determining a second hand position at which the hand gesture was completed;

causing haptic feedback to be generated in response to detecting an intersection between the at least one hand position and a virtual object associated with an audio event included in an audio stream;

detecting the hand gesture based on the at least one hand position;

in response to the hand gesture, modifying, via a processor, a spatial audio parameter associated with the audio event to generate a modified audio stream by;

moving the audio event from a first location in a sound space associated with the first hand position towards a second location in the sound space associated with the second hand position, and

determining, via a physics model, an interaction between a first virtual object that corresponds to the audio event and a second virtual object that corresponds to the hand of the user; and

causing the modified audio stream to be reproduced for output to the user.

20. A system for modifying an audio parameter based on a gesture, comprising:

a head-mounted device;

one or more sensors included in the head-mounted device and configured to acquire sensor data associated with a hand of a user;

a processor coupled to the one or more sensors and configured to;

analyze the sensor data to determine at least one hand position;

cause haptic feedback to be generated in response to detecting an intersection between the at least one hand position and a virtual object associated with an audio event included in an audio stream;

detect a hand gesture based on the at least one hand position; and

in response to the hand gesture, move a first audio event included in an audio stream from a first location in a sound space towards a second location in the sound space based on a head-related transfer function to generate a modified audio stream; and

speakers coupled to the processor and configured to reproduce the modified audio stream.