INTERACTIVE CONTENT CREATION

• US 20120165964A1
• Filed: 12/27/2010
• Published: 06/28/2012
• Est. Priority Date: 12/27/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Flaks, Jason, Poot, Rudy Jacobus, Kipman, Alex Aben-Athar, Miles, Chris, Fuller, Andrew John, Margolis, Jeffrey Neil
• Granted Patent: US 9,123,316 B2
• Time in Patent Office: Days
• US Class Current: 700/94
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/455 Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data

Abstract

An audio/visual system (e.g., such as an entertainment console or other computing device) plays a base audio track, such as a portion of a pre-recorded song or notes from one or more instruments. Using a depth camera or other sensor, the system automatically detects that a user (or a portion of the user) enters a first collision volume of a plurality of collision volumes. Each collision volume of the plurality of collision volumes is associated with a different audio stem. In one example, an audio stem is a sound from a subset of instruments playing a song, a portion of a vocal track for a song, or notes from one or more instruments. In response to automatically detecting that the user (or a portion of the user) entered the first collision volume, the appropriate audio stem associated with the first collision volume is added to the base audio track or removed from the base audio track.

Claims

1. A method for audio content creation, comprising:

playing audio content;

automatically tracking movement of a user, including automatically detecting predefined motion of the user; and

automatically changing the audio content being played in response to detecting the predefined motion of the user.

2. The method of claim 1, wherein:

the predefined motion of the user includes a portion of the user entering a first collision volume;

the automatically changing the audio content includes automatically changing the audio content being played in response to detecting that the portion of the user entered the first collision volume;

the playing audio content includes playing a base audio track; and

the automatically changing the audio content being played includes adding a first audio stem to the base track and synchronized with the base track in response to detecting that the portion of the user entered the first collision volume, the first audio stem is associated with the first collision volume.

3. The method of claim 2, further comprising:

automatically detecting that a portion of the user entered a second collision volume that is different than the first collision volume, the first collision volume and the second collision volume are part of a plurality of collision volumes, each collision volume of the plurality is associated with a different audio stem, a second audio stem is associated with the second collision volume; and

automatically changing the audio content being played by adding the second audio stem to the base track in response to detecting that the portion of the user entered the second collision volume.

4. The method of claim 3, further comprising:

automatically detecting that a portion of the user entered the second collision volume after changing the audio content being played by adding the second audio stem to the base track; and

automatically changing the audio content being played by removing the second audio stem from the base track in response to detecting that the portion of the user entered the second collision volume after changing the audio content being played by adding the second audio stem to the base track.

5. The method of claim 2, further comprising:

detecting predefined movement within the first collision volume; and

performing an effect on the audio content being played in response to detecting the predefined movement.

6. The method of claim 2, further comprising:

automatically detecting a location of the user, the base audio track is chosen from multiple audio tracks based on the detected location of the user.

7. The method of claim 2, wherein:

the base audio track is a portion of a song; and

the first audio stem is sound from a subset of instruments playing the song.

8. The method of claim 2, wherein:

the base audio track is pre-recorded sound from a first instrument; and

the first audio is pre-recorded sound from a second instrument.

9. The method of claim 2, wherein:

the base audio track is first sound from a synthesizer; and

the first audio is second sound from the synthesizer.

10. The method of claim 2, further comprising:

automatically detecting which zone the user is located in of a plurality of zones, each zone corresponds to a different base audio track, the playing the base audio track is performed in response to the automatically detecting which zone the user is located in.

11. The method of claim 1, further comprising:

displaying visual content and changing the visual content in response to detecting the predefined motion of the user, the visual content are simultaneously generated by both the movements of the user and analysis of the music itself.

12. The method of claim 1, wherein:

the automatically tracking movement of the user is performed using a depth camera.

13. The method of claim 1, wherein:

the automatically tracking movement of the user includes automatically tracking movement of multiple users; and

the automatically changing the audio content being played includes automatically changing the audio content being played in response to and based on the tracked movement of multiple users such that different movement changes the audio content being played in different ways.

16. The apparatus of claim 13, wherein:

the first predefined movement includes a portion of the user entering a first collision volume of a plurality of collision zone; and

the processor detects predefined motion in a collision volume using data from the depth camera and performs an audio effect on audio being played in response to the detecting of the predefined motion in the collision volume.

17. The apparatus of claim 13, wherein:

the predefined movement includes any of a gesture, motion or a center of mass, movement of a limb.

14. An apparatus that creates audio content, comprising:

a depth camera;

a display interface;

an audio interface; and

a processor in communication with the depth camera, display interface and audio interface, the processor plays a first base audio track, the processor is programmed to automatically detect first predefined movement of a user from a plurality of predefined movements based on data from the depth camera, each predefined movement is associated with a different audio stem, the processor adds a first audio stem to the base track in response to detecting that the user performed the first predefined movement, the first audio stem corresponds to the first predefined movement.

15. The apparatus of claim 14, wherein:

the processor is programmed to automatically detect that a portion of a user performs a second predefined movement based on data from the depth camera, the processor adds a second audio stem to the base track that in response to detecting that the user performed the second predefined movement, the second base track corresponds to the second collision volume.

18. One or more processor readable storage devices storing processor readable code thereon, the processor readable code for programming one or more processors to perform a method comprising:

defining one or more base tracks for a plurality of movement zones;

identifying audio stems for a set of collision volumes for each zone; and

creating code based on the defined one or more base tracks for the plurality of movement zones and the indentified audio stems for the set of collision volumes for each zone, the code capable of configuring a computing device to play the one or base tracks depending on which zone a user is positioned, the code capable of configuring the computing device to add or subtract audio stems based on the user intersecting corresponding collision volumes.

19. One or more processor readable storage devices according to claim 18, wherein:

the method further comprises associating one or more audio effects with one or more predefined movements within one or more of the collision volumes; and

the code is capable of configuring the computing device to perform the one or more audio effects in response to one or more predefined movements within one or more of the collision volumes.

20. One or more processor readable storage devices according to claim 18, wherein:

the method further comprises defining a custom avatar; and

the creating code includes providing a description of the avatar in the code.