DEVICE AND METHOD FOR INTERPRETING MUSICAL GESTURES

• US 20120062718A1
• Filed: 02/12/2010
• Published: 03/15/2012
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: Commissariat a L'Energie Atomique (AES Corporation), Movea SA (TDK Corporation)
• Original Assignee: Commissariat a L'Energie Atomique (AES Corporation), Movea SA (TDK Corporation)
• Inventors: David, Dominique
• Granted Patent: US 9,171,531 B2
• Time in Patent Office: Days
• US Class Current: 348/77
• CPC Class Codes: G10H 1/18 Selecting circuits
• CPC Class Codes: G10H 1/40 Rhythm
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10H 2210/385 Speed change, i.e. variations from preestablished tempo, tempo change, e.g. faster or slower, accelerando or ritardando, without change in pitch with repetitive changes in pitch, e.g. scratch DJ effects G10H2210/241
• CPC Class Codes: G10H 2220/135 Musical aspects of games or videogames; Musical instrument-shaped game input interfaces game background music G10H2210/026; musical game scoring or performance evaluation G10H2210/091
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/206 Conductor baton movement detection used to adjust rhythm, tempo or expressivity of, e.g. the playback of musical pieces
• CPC Class Codes: G10H 2220/395 Acceleration sensing or accelerometer use, e.g. 3D movement computation by integration of accelerometer data, angle sensing with respect to the vertical, i.e. gravity sensing. conductor baton movement sensing G10H2220/206, angle sensing without reference to gravity G10H2220/391; player body joint angle sensing G10H2220/321

Abstract

Musical rendition is provided through the use of microsensors, in particular of accelerometers and magnetometers or rate gyros, and through an appropriate processing of the signals from the microsensors. In particular, the processing uses a merging of the data output from the microsensors to eliminate false alarms in the form of movements of the user unrelated to the music. The velocity of the musical strikes is also measured. Embodiments make it possible to control the running of mp3 or wav type music files to be played back.

Claims

1. A device for interpreting gestures of a user comprising:

at least one input module for measurements comprising at least one motion capture assembly on at least a first and a second axis,

a module for processing signals sampled at the output of the input module, and

an output module capable of playing back the musical meaning of said gestures,

the signal processing module comprising a submodule for analyzing and interpreting gestures comprising a filtering function, a function for detecting meaningful gestures by comparison of the variation between at least two successive values in the sample of at least one of the signals originating from at least the first axis of the set of sensors with at least a first selected threshold value and a function for confirming the detection of a meaningful gesture,

wherein said function for confirming the detection of a meaningful gesture is capable of comparing at least one of the signals originating from at least the second axis of the set of sensors with at least a second selected threshold value.

2. The device for interpreting gestures of claim 1, wherein the filtering function is executable by at least one pair of two successive low-pass recursive filters configured to receive as input at least one of the signals output from the module (10).

3. The device for interpreting gestures of claim 2, wherein the function for detecting meaningful gestures is configured to identify changes of sign between two successive values in the sample of the difference between at least one output from the first filter of at least one of the pairs of filters at the current value and at least one output from the second filter of the same pair of filters for the same signal at the preceding value.

4. The device for interpreting gestures of claim 3, wherein the submodule for analyzing and interpreting gestures also comprises a function for measuring the velocity of the gesture detected at the output of the detection confirmation function.

5. The device for interpreting gestures of claim 4, wherein the function for measuring velocity is capable of computing the travel (Max−

Min) between two detected meaningful gestures.

6. The device for interpreting gestures of claim 3, wherein the second filter is capable of operating at a cut-off frequency less than that of the first filter.

7. The device for interpreting gestures of claim 2, wherein the input module comprises at least a first sensor of accelerometer type and a second sensor chosen from the group of sensors of magnetometer and rate gyro types.

8. The device for interpreting gestures of claim 7, wherein the function for detecting meaningful gestures is capable of receiving as input at least one output from the second recursive filter of one of the pairs of filters applied to at least one of the signals from the first sensor.

9. The device for interpreting gestures of claim 7, wherein the function for confirming the detection of a meaningful gesture is capable of receiving as input at least one output from the second recursive filter of one of the pairs of filters applied to at least one of the signals from the second sensor.

10. The device for interpreting gestures of claim 9, wherein the threshold selected for the function for confirming the detection of a meaningful gesture is of the order of 5/1000 as a relative value of the filtered signal.

11. The device for interpreting gestures of claim 4, wherein the input module receives the signals from at least two sensors positioned on two independent parts of the body of the user, a first sensor supplying, via one of the pairs of recursive filters, a signal as input for the function for detecting meaningful gestures and a second sensor supplying, via one of the pairs of recursive filters, a signal as input for the function for measuring the velocity of the gesture detected at the output of the function for confirming the detection of a meaningful gesture.

12. The device for interpreting gestures of claim 1, wherein the signal processing module comprises an input submodule for prerecorded multimedia contents.

13. The device for interpreting gestures of claim 12, wherein the input submodule for multimedia contents comprises a function for partitioning said multimedia contents into time windows that can be used to perform a second confirmation of detection of the detected meaningful gestures.

14. The device for interpreting gestures of claim 1, wherein the input module is capable of transmitting to the processing module a signal representative of the position of the user in a plane substantially orthogonal to the direction of the detected meaningful gesture to perform a second confirmation thereof.

15. The device for interpreting gestures of claim 1, wherein the output module comprises a submodule for playing back a prerecorded file of signals to be played back and the processing module comprises a submodule for controlling the timing of said prerecorded signals, said playback submodule being able to be programmed to determine the times at which strikes controlling the runrate of the file are expected, and said timing control submodule is capable of computing, for a certain number of control strikes, a relative corrected speed factor of preprogrammed strikes in the playback submodule and strikes actually entered in the timing control submodule and a relative intensity factor of the velocities of said strikes actually entered and expected then of adjusting the runrate of said timing control submodule to adjust said corrected speed factor on the subsequent strikes to a selected value and the intensity of the signals output from said playback submodule according to said relative intensity factor of the velocities.

16. The device of claim 15, wherein the velocity of the entered strike is computed on the basis of the deviation of the signal output from the second sensor.

17. The device for interpreting gestures of claim 1, wherein the input module comprises a submodule capable of interpreting gestures of the user whose output is used by the timing control submodule to control a characteristic of the audio output selected from the group consisting of vibrato and tremolo.

18. The device for interpreting gestures of claim 15, wherein the playback submodule comprises a function for placing tags in the file of prerecorded signals to be played back at times at which strikes controlling the runrate of the file are expected, said tags being generated automatically according to the rate of the prerecorded signals and being able to be shifted by a MIDI interface.

19. The device for interpreting gestures of claim 15, wherein the value selected in the timing control submodule to adjust the running speed of the playback submodule is equal to a value selected from a set of computed values of which one of the limits is computed by application of a corrected speed factor equal to the ratio of the time interval between the next tag and the preceding tag minus the time interval between the current strike and the preceding strike to the time interval between the current strike and the preceding strike and whose other values are computed by linear interpolation between the current value and the value corresponding to that of the limit used for the application of the corrected speed factor.

20. The device for interpreting gestures of claim 19, wherein the value selected in the timing control submodule to adjust the running speed of the playback submodule is equal to the value corresponding to that of the limit used for the application of the corrected speed factor.

21. A method for interpreting meaningful gestures of a user comprising at least one step for inputting measurements originating from at least one motion capture assembly along at least a first and a second axis, a step for processing signals sampled at the output of the input step and an output step capable of playing back the musical meaning of said gestures, the signal processing step comprising a substep for analyzing and interpreting gestures comprising at least one filtering step, a function for detecting meaningful gestures by comparison of the variation between two successive values in the sample of at least one of the signals originating from at least the first axis of the set of sensors with at least a first selected threshold value and a function for confirming the detection of a meaningful gesture, wherein said function for confirming the detection of a meaningful gesture is capable of comparing at least one of the signals originating from at least the second axis of the set of sensors with at least a second selected threshold value.

22. The method for interpreting gestures of claim 21, wherein the output step comprises a substep for playing back a prerecorded file of signals to be played back and in that the processing step comprises a substep for controlling the timing of said prerecorded signals, said playback substep being capable of determining the times at which strikes controlling the runrate of the file are expected, and said timing control substep being capable of computing, for a certain number of control strikes, a relative corrected speed factor of preprogrammed strikes in the playback substep and of strikes actually entered during the timing control substep and a relative intensity factor of the velocities of said strikes actually entered and expected then of adjusting the runrate of said prerecorded file to adjust said corrected speed factor on the subsequent strikes to a selected value and the intensity of the signals output from the playback step according to said relative intensity factor of the velocities.