Interactive content creation

• US 9,123,316 B2
• Filed: 12/27/2010
• Issued: 09/01/2015
• Est. Priority Date: 12/27/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Flaks, Jason, Poot, Rudy Jacobus, Kipman, Alex Aben-Athar, Miles, Chris, Fuller, Andrew John, Margolis, Jeffrey Neil
• Primary Examiner(s): Lao, Lun-See
• Application Number:
  US12/978,799
• Publication Number:
  US 20120165964A1
• Time in Patent Office: 1,709 Days
• Field of Search: 700/94, 381104-107
• US Class Current: 1/1
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/455 Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data

Abstract

An audio/visual system (e.g., such as an entertainment console or other computing device) plays a base audio track, such as a portion of a pre-recorded song or notes from one or more instruments. Using a depth camera or other sensor, the system automatically detects that a user (or a portion of the user) enters a first collision volume of a plurality of collision volumes. Each collision volume of the plurality of collision volumes is associated with a different audio stem. In one example, an audio stem is a sound from a subset of instruments playing a song, a portion of a vocal track for a song, or notes from one or more instruments. In response to automatically detecting that the user (or a portion of the user) entered the first collision volume, the appropriate audio stem associated with the first collision volume is added to the base audio track or removed from the base audio track.

Claims

1. A method for audio content creation, comprising:

defining a plurality of three-dimensional collision volumes, the plurality of three-dimensional collision zones being at least partially different three-dimensional spaces;

assigning a base track of music to each of the plurality of three-dimensional collision volumes;

playing audio content;

automatically tracking movement of a user, including automatically detecting predefined motion of the user in one or more of the plurality of collision volumes; and

automatically changing the audio content upon entry of a body part into one or more of the collision volumes, the change in the audio content being based on which collision volume or volumes are entered by a body part and what predefined motions are performed within the one or more collision volumes.

2. The method of claim 1, wherein:

the predefined motion of the user includes a portion of the user entering a first collision volume of the one or more collision volumes;

the automatically changing the audio content includes automatically changing the audio content being played in response to detecting that the portion of the user entered the first collision volume;

the playing audio content includes playing a base audio track; and

the automatically changing the audio content being played includes adding a first audio stem to the base track and synchronized with the base track in response to detecting that the portion of the user entered the first collision volume, the first audio stem is associated with the first collision volume.

3. The method of claim 2, further comprising:

automatically detecting that a portion of the user entered a second collision volume of the one or more collision volumes, the second collision volume being different than the first collision volume, the first collision volume and the second collision volume are part of a plurality of collision volumes, each collision volume of the plurality is associated with a different audio stem, a second audio stem is associated with the second collision volume; and

automatically changing the audio content being played by adding the second audio stem to the base track in response to detecting that the portion of the user entered the second collision volume.

4. The method of claim 3, further comprising:

automatically detecting that a portion of the user entered the second collision volume after changing the audio content being played by adding the second audio stem to the base track; and

automatically changing the audio content being played by removing the second audio stem from the base track in response to detecting that the portion of the user entered the second collision volume after changing the audio content being played by adding the second audio stem to the base track.

5. The method of claim 2, further comprising:

detecting predefined movement within the first collision volume; and

performing an effect on the audio content being played in response to detecting the predefined movement.

6. The method of claim 2, further comprising:

automatically detecting a location of the user, the base audio track is chosen from multiple audio tracks based on the detected location of the user.

7. The method of claim 2, wherein:

the base audio track is a portion of a song; and

the first audio stem is sound from a subset of instruments playing the song.

8. The method of claim 2, wherein:

the base audio track is pre-recorded sound from a first instrument; and

the first audio is pre-recorded sound from a second instrument.

9. The method of claim 2, wherein:

the base audio track is first sound from a synthesizer; and

the first audio is second sound from the synthesizer.

10. The method of claim 2, further comprising:

automatically detecting which zone the user is located in of a plurality of zones, each zone corresponds to a different base audio track, the playing the base audio track is performed in response to the automatically detecting which zone the user is located in.

11. The method of claim 1, further comprising:

displaying visual content and changing the visual content in response to detecting the predefined motion of the user, the visual content are simultaneously generated by both the movements of the user and analysis of the music itself.

12. The method of claim 1, wherein:

the automatically tracking movement of the user is performed using a depth camera.

13. The method of claim 1, wherein:

the automatically tracking movement of the user includes automatically tracking movement of multiple users; and

the automatically changing the audio content being played includes automatically changing the audio content being played in response to and based on the tracked movement of multiple users such that different movement changes the audio content being played in different ways.

14. One or more processor readable storage devices storing processor readable code thereon, the processor readable code for programming one or more processors to perform a method comprising:

defining one or more base tracks for a plurality of movement zones;

identifying audio stems for a set of collision volumes for each zone; and

creating code based on the defined one or more base tracks for the plurality of movement zones and the identified audio stems for the set of collision volumes for each zone, the code capable of configuring a computing device to automatically play the one or base tracks depending on which zone a user is positioned, the code capable of configuring the computing device to add or subtract audio stems based on the user intersecting corresponding collision volumes.

15. One or more processor readable storage devices according to claim 14, wherein:

the method further comprises associating one or more audio effects with one or more predefined movements within one or more of the collision volumes; and

the code is capable of configuring the computing device to perform the one or more audio effects in response to one or more predefined movements within one or more of the collision volumes.

16. One or more processor readable storage devices according to claim 14, wherein:

the method further comprises defining a custom avatar; and

the creating code includes providing a description of the avatar in the code.