Head pose mixing of audio files

• US 10,681,489 B2
• Filed: 09/13/2016
• Issued: 06/09/2020
• Est. Priority Date: 09/16/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Mangiat, Stephen Vincent, Tucker, Michael Benson, Tajik, Anastasia Andreyevna
• Primary Examiner(s): Badawi, Angie
• Application Number:
  US15/264,164
• Publication Number:
  US 20170078825A1
• Time in Patent Office: 1,365 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10H 2210/131 Morphing, i.e. transformation of a musical piece into a new different one, e.g. remix
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/40 Visual indication of stereophonic sound image

Abstract

Examples of wearable devices that can present to a user of the display device an audible or visual representation of an audio file comprising a plurality of stem tracks that represent different audio content of the audio file are described. Systems and methods are described that determine the pose of the user; generate, based on the pose of the user, an audio mix of at least one of the plurality of stem tracks of the audio file; generate, based on the pose of the user and the audio mix, a visualization of the audio mix; communicate an audio signal representative of the audio mix to the speaker; and communicate a visual signal representative of the visualization of the audio mix to the display.

Claims

1. A wearable device comprising:

non-transitory memory configured to store an audio file comprising a plurality of stem tracks, with each stem track representing different audio content of the audio file, wherein at least one of the plurality of stem tracks comprises animation metadata related to audio content of the at least one of the plurality of stem tracks;

a sensor configured to measure information associated with a pose of a user of the wearable device;

a display configured to present images to an eye of the user of the wearable device;

a speaker configured to present sounds to the user of the wearable device; and

a processor in communication with the non-transitory memory, the sensor, the speaker, and the display, the processor programmed with executable instructions to;

determine the pose of the user, the pose indicating at least a pose direction the user is facing in a three-dimensional (3D) environment;

cause to display a virtual object comprising a plurality of surfaces associated with different stem tracks of the audio file;

determine a change in an orientation of the virtual object relative to the pose of the user in response to a user movement relative to the virtual object within the 3D environment or a user indication to reorient the virtual object;

update, based at least partly on the pose of the user and the change in the orientation of the virtual object, display of the virtual object in the 3D environment;

identify one or more surfaces of the virtual object along the pose direction of the user in the 3D environment;

select one or more stem tracks associated with the identified one or more surfaces to present through the speaker;

determine an audio mix of the selected one or more stem tracks; and

communicate an audio signal representative of the audio mix to the speaker.

2. The wearable device of claim 1, wherein stem tracks of the plurality of stem tracks are associated with different musical instruments or a vocal track.

3. The wearable device of claim 1, wherein the stem tracks of the plurality of stem tracks comprise information associated with a spatial location and a spatial directionality of the stem track.

4. The wearable device of claim 1, wherein the sensor comprises at least one of:

gyroscope, an accelerometer, an inertial measurement unit, an eye tracking camera, or a camera configured to image an environment of the user.

5. The wearable device of claim 1, wherein the pose comprises at least one of:

a head pose, an eye pose, a body pose, or a hand gesture.

6. The wearable device of claim 1, wherein the processor is further programmed to cause to display the virtual object in the changed orientation.

7. The wearable device of claim 6, wherein the virtual object comprises a graphical representation of an instrument or a person associated with performance of the stem track.

8. The wearable device of claim 6, wherein at least some of the virtual object is presented at different depth planes relative to the user.

9. The wearable device of claim 1, wherein the wearable device is configured to recognize a physical object in an environment of the user, and to present a graphical representation of a stem track of the plurality of stem tracks as being associated with the physical object.

10. The wearable device of claim 1, wherein the wearable device is configured to receive input from a user input device, and the processor is configured to generate the audio mix or the virtual object based at least in part on the input.

11. The wearable device of claim 1, wherein the animation metadata comprises information relating to one or more of:

movement of a musical performer associated with the audio file, acoustics of a performance venue, or size of the performance venue.

12. The wearable device of claim 1, wherein the processor is further configured to receive an input to change the orientation of the virtual object.

13. The wearable device of claim 1, wherein the audio mix comprises a pre-mixed combination of the plurality of stem tracks.

14. A method for interacting with an audio file, the method comprising:

under control of a wearable device comprising hardware computer processor, a sensor configured to measure information associated with a pose of a user of the wearable device;

a display configured to present images to an eye of the user of the wearable device;

a speaker configured to present sounds to the user of the wearable device;

accessing a plurality of stem tracks associated with an audio file, wherein at least one of the plurality of stem tracks comprises;

animation metadata related to audio content of the audio file;

detecting the pose of the user, the pose indicating at least a pose direction the user is facing in a three-dimensional (3D) environment;

causing to display a virtual object comprising a plurality of surfaces associated with different stem tracks of the audio file;

determining a change in orientation of the virtual object relative to the pose of the user in response to a user movement relative to the virtual object within the 3D environment or a user indication to reorient the virtual object;

updating, based at least partly on the pose of the user and the change in the orientation of the virtual object, display of the virtual object in the 3D environment;

identifying one or more surfaces of the virtual object along the pose direction of the user in the 3D environment;

selecting one or more stem tracks of the identified one or more surfaces to present through the speaker;

determining an audio mix of the selected one or more stem tracks; and

communicating an audio signal representative of the audio mix to the speaker.

15. The method of claim 14, wherein stem tracks of the plurality of stem tracks are associated with different musical instruments or a vocal track.

16. The method of claim 14, wherein the selected one or more stem tracks comprises information associated with a spatial location and a spatial directionality of the selected one or more stem track.

17. The method of claim 14, further comprising causing to display the virtual object in the changed orientation.

18. The method of claim 17, wherein the virtual object comprises a representation of musical instruments associated with the respective stem tracks or persons playing the musical instruments associated with the respective stem tracks.

19. The method of claim 17, wherein at least some of the virtual object is presented at different depth planes relative to the user.

20. The method of claim 14, wherein the pose comprises at least one of:

a head pose, an eye pose, a body pose, or a hand gesture.

21. The method of claim 14, comprising updating the virtual object by emphasizing or de-emphasizing at least one portion of the virtual object associated with a directional stem track.

22. The method of claim 14, further comprising:

receiving a selection of a stem track of the plurality of stem tracks;

receiving an indication to modify the stem track, wherein the indication comprises a change in the pose of the user or an input from a user input device; and

modifying the stem track based at least partly on the indication.

23. The method of claim 22, where in modifying the stem track comprising at least one of:

adjusting volume, replacing the stem track with another stem track, muting the stem track, emphasizing the stem track over other stem tracks of the plurality of stem tracks.

24. The method of claim 14, wherein the animation metadata comprises information relating to one or more of:

movement of a musical performer associated with the audio file, acoustics of a performance venue, or size of the performance venue.

25. The method of claim 14, wherein the processor is further configured to receive an input to change the orientation of the virtual object.

26. The method of claim 14, wherein the audio mix comprises a pre-mixed combination of the plurality of stem tracks.