Interactive music playback system utilizing gestures

• US 20030159567A1
• Filed: 10/18/2002
• Published: 08/28/2003
• Est. Priority Date: 10/18/2002
• Status: Abandoned Application

• Current Assignee: Morton Subotnick
• Original Assignee: Morton Subotnick
• Inventors: Subotnick, Morton
• Application Number:
  US10/258,059
• Publication Number:
  US 20030159567A1
• Time in Patent Office: Days
• US Class Current: 84/626
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2220/161 with 2D or x/y surface coordinates sensing graphical user interface or touchscreen input G10H2220/091; microtonal keyboard G10H2210/401
• CPC Class Codes: G10H 2240/056 MIDI or other note-oriented file format
• CPC Class Codes: G10H 2240/061 MP3, i.e. MPEG-1 or MPEG-2 Audio Layer III, lossy audio compression
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/135 Library retrieval index, i.e. using an indexing scheme to efficiently retrieve a music piece

Abstract

An interactive music system (10) in accordance with various aspects of the invention lets a user control the playback of recorded music according to gestures entered via an input device (14), such as a mouse. The system includes modules which interpret input gestures made on a computer input device and adjust the playback of audio data in accordance with input gesture data. Various methods for encoding sound information in an audio data produce with meta-data indicating how it can be varied during playback are also disclosed. More specifically, a gesture input system receives user input from a device, such as a mouse, and interprets this data as one of a number of predefined gestures which are assigned an emotional or interpretive meaning according to a “character” hierarchy or library (16) of gesture descriptions. The received gesture inputs are used to alter the character of music which is being played in accordance with the meaning of the gesture. For example, an excited gesture can effect the playback in one way, while a quiet playback may affect it in another. The specific result is a combination of the gesture made by the user, its interpretation by the computer, and a determination of how the interpreted gesture should effect the playback. Entry of a excited gesture thus may brighten the playback, e.g., by changing increasing the tempo, changing from a minor to major key, varying the instruments used or the style in which they are played, etc. In addition, the effects can be cumulative, allowing a user to progressively alter the playback. To further enhance the interactive nature of the system, users can be given the ability to alter the effect of a given gesture or assign a gesture to specific places in the character hierarchy.

Claims

1. An interactive music method comprising the steps of:

receiving a gesture;

interpreting the gesture in accordance with a plurality of predefined gestures;

assigning an emotional meaning to the gesture; and

playing music according to the assigned emotional meaning.

2. The method of claim 1 wherein the gesture is interpreted by scaling the gesture into a value indicating a group of parameters selected from the group consisting of bentness, jerkiness and length of the gesture.

3. The method of claim 1 wherein the gesture is received by an input device selected from the group consisting of a mouse, joystick, trackball, tablet, data gloves, electronic conducting baton, video motion tracking device, blood pressure tracking device, heart rate tracking device and muscle tracking device.

4. The method of claim 3 further comprising the steps of:

calculating a duration of time between when the mouse is up and when the mouse is down;

calculating a number of pixels traveled by the mouse;

calculating variations in a velocity of the mouse within the gesture; and

calculating an arm of the mouse movement throughout the gesture.

5. The method of claim 2 further comprising the steps of:

calculating a number and location of horizontal and vertical direction changes in the gesture; and

determining a bentness of the gesture according to the calculated number and location.

6. The method of claim 5 further comprising the step of scaling the bentness with reference to a number of bends per unit length.

7. The method of claim 1 further comprising the step of valuing the received gesture according to a three-tier scale including little bentness, medium bentness and very bent.

8. The method of claim 1 further comprising the step of valuing the received gesture according to a three-tier scale including little jerkiness, some jerkiness and very jerky.

9. The method of claim 2 further comprising the steps of:

mapping the parameters to the predefined gestures; and

associating the mapped parameters with corresponding emotional meanings.

10. The method of claim 1 wherein the interpreting step includes different levels of responsiveness to the received gesture.

11. The method of claim 10 further comprising the step of adjusting the responsiveness to the received gesture.

12. The method of claim 10 wherein the levels of responsiveness comprise a DJ mode and a simple composition mode.

13. The method of claim 1 further comprising the steps of:

storing a plurality of musical segments in database;

associating the musical segments with the predefined gestures;

selecting one of the musical segments according to the emotional meaning assigned to the received gesture; and

playing the selected musical segment.

14. The method of claim 13 further comprising the steps of:

randomly selecting one of the musical segments corresponding to the emotional meaning; and

playing the randomly selected musical segment.

15. The method of claim 13 further comprising the step of playing a predefined sequence if more than one of the musical segments correspond to the assigned emotional meaning.

16. An interactive music system comprising:

a receiver receiving a gesture;

an interpreter device interpreting the gesture in accordance with a plurality of predefined gestures;

an assignor device assigning an emotional meaning to the gesture; and

a playback device playing music according to the assigned emotional meaning.

17. The system of claim 16 wherein the gesture is interpreted by scaling the gesture into a value indicating a group of parameters selected from the group consisting of bentness, jerkiness and length of the gesture.

18. The system of claim 16 wherein the gesture is received by an input device selected from the group consisting of a mouse, joystick, trackball, tablet data gloves, electronic conducting baton, video motion tracking device, blood pressure tracking device, heart rate tracking device and muscle tracking device.

19. The system of claim 18 further comprising:

a calculator calculating a duration of time between when the mouse is up and when the mouse is down, calculating a number of pixels traveled by the mouse, calculating variations in a velocity of the mouse within the gesture, and calculating an arm of the mouse movement throughout the gesture.

20. The system of claim 17 further comprising:

a calculator calculating a number and location of horizontal and vertical direction changes in the gesture; and

the system determining a bentness of the gesture according to the calculated number and location.

21. The system of claim 20 further comprising a scalar device scaling the bentness with reference to a number of bends per unit length.

22. The system of claim 16 wherein the system values the received gesture according to a three-tier scale including little bentness, medium bentness and very bent.

23. The system of claim 16 wherein the system values the received gesture according to a three-tier scale including little jerkiness, some jerkiness and very jerky.

24. The system of claim 17 further comprising:

a mapper mapping the parameters to the predefined gestures; and

the system associating the mapped parameters with corresponding emotional meanings.

25. The system of claim 16 wherein the interpreter device interprets the gesture according to different levels of responsiveness to the received gesture.

26. The system of claim 25 further comprising an adjustor device adjusting the responsiveness to the received gesture.

27. The system of claim 25 wherein the levels of responsiveness comprise a DJ mode and a simple composition mode.

28. The system of claim 16 further comprising:

a database storing a plurality of musical segments wherein the musical segments are associated with the predefined gestures; and

a selector device selecting one of the musical segments according to the emotional meaning assigned to the received gesture wherein the playback device plays the selected musical segment.

29. The system of claim 28 further comprising a random selector randomly selecting one of the musical segments corresponding to the emotional meaning wherein the playback device plays the randomly selected musical segment.

30. The system of claim 28 wherein a predefined sequence is played if more than one of the musical segments correspond to the assigned emotional meaning.