Method and apparatus for controlling electronic tone generation in accordance with a detected type of performance gesture

• US 5,288,938 A
• Filed: 12/05/1990
• Issued: 02/22/1994
• Est. Priority Date: 12/05/1990
• Status: Expired due to Term

• Current Assignee: Yamaha Corporation
• Original Assignee: Yamaha Corporation
• Inventors: Wheaton, James A.
• Primary Examiner(s): Shoop, Jr., William M.
• Assistant Examiner(s): Sircus, Brian
• Application Number:
  US07/622,517
• Time in Patent Office: 1,175 Days
• Field of Search: 84/DIG. 12, 84/723-725, 84/730, 84/742-745, 84/603, 84/615, 84/626, 84/600, 84/658, 84/734, 382/1, 382/4, 382/8, 382/16, 382/25, 382/28
• US Class Current: 84/600
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10H 1/055 by switches with variable impedance elements
• CPC Class Codes: G10H 1/34 Switch arrangements, e.g. keyboards or mechanical switches specially adapted for electrophonic musical instruments
• CPC Class Codes: G10H 2220/161 with 2D or x/y surface coordinates sensing
• CPC Class Codes: G10H 2230/305 Spint drum snare, i.e. mimicking using strands of snares made of curled metal wire, metal cable, plastic cable, or gut cords stretched across the drumhead, e.g. snare drum, side drum, military drum, field drum

Abstract

A method and apparatus for controlling electronic tone generation determines a type of gesture used by a player to contact a performance surface provided with an array of pressure sensors. The performance surface repeatedly is scanned to determine which of the sensors has been activated at each scan time. When a player contacts the performance surface, parameter data associated with the incoming gesture is compared with gesture recognition rules corresponding to various reference gestures. The gesture recognition rules may be preset, and/or the control system may be provided with an optional programming capability which allows the player to establish individualized gesture recognition rules in accordance with reference gestures input by the player. An electronic tone generator is controlled in accordance with the reference gesture which is determined to best fit an incoming gesture. Furthermore, the electronic tone generator may be controlled to generate a tone having a particular tone characteristic determined in accordance with a corresponding parameter of a particular gesture type such that a player may achieve a timbral variation simply by modifying the corresponding parameter of the particular reference gesture.

Claims

1. A method of controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a performance surface, comprising the steps of:

providing a performance surface including an array of pressure sensors, each of which provides an output;

analyzing the output from each sensor of the array to determine a pattern of plural sensors which have been substantially simultaneously activated by a player, said pattern corresponding to a type of gesture used by a player to contact the performance surface; and

controlling an electronic tone generator in accordance with a type of gesture determined to have been used by the player to contact the performance surface.

2. The method of claim 1, wherein the performance surface is a drum-like surface.

3. The method of claim 1, wherein the electronic tone generator controlled is a sampling type tone generator comprising a memory storing various waveshapes corresponding to various tones, and wherein one of said waveshapes is selected based on the type of gesture determined to have been used, and the tone generator is controlled to generate a tone in accordance with the selected waveshape.

4. The method of claim 1, wherein the tone generator is a tone synthesizer which controls at least one of pitch and envelope in accordance with a detected gesture.

5. The method of claim 1, wherein the electronic tone generator is controlled to generate a tone having at least one tone characteristic which is determined in accordance with at least one corresponding parameter of a particular gesture type when the particular gesture type is determined to have been used by the player, whereby the player may achieve a timbral variation simply by modifying the corresponding parameter of the particular gesture type.

6. The method of claim 5, wherein the tone characteristic is one of volume, pitch, modulation, and harmonic content.

7. The method of claim 5, wherein the parameter is one of average pressure value for the particular gesture type and distance from the center of the performance surface to the center of the gesture.

8. A method of controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a pressure sensitive surface, comprising the steps of:

providing a pressure sensitive surface including an array of pressure sensors to detect a type of gesture used by a player to contact the performance surface, the array of pressure sensors respectively detecting pressures applied thereto;

detecting the substantially simultaneous activation of plural ones of the pressure sensors to which at least a threshold amount of pressure is applied when a player contacts the surface;

determining a type of gesture used by a player to contact the surface in accordance with the detected activation of said plural pressure sensors; and

controlling an electronic tone generator in accordance with a type of gesture determined to have been used by the player to contact the surface.

9. A method of controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a performance surface, comprising the steps of:

providing a performance surface comprising an array of pressure sensors, each of which generates an output signal when activated;

detecting the output of each sensor of the array of sensors;

determining a type of gesture used by the player to contact the array in accordance with the detected output of plural sensors substantially simultaneously activated by a players; and

controlling an electronic tone generator in accordance with a type of gesture determined to have been used by the player to contact the array.

10. A method of controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a performance surface, comprising the steps of:

providing a performance surface comprising an array of pressure sensors, each of which generates an output signal representative of an amount of pressure applied by the player;

detecting a position of each sensor based upon the sensor output;

detecting an amount of pressures applied to each sensor based upon the sensor output;

determining a pressure pattern of plural activated sensors corresponding to the type of gesture used by a player to contact the array in accordance with the detected position and the detected amount of pressure for each sensor output; and

controlling an electronic tone generator in accordance with a type of gesture determined to have been used by the player to contact the array.

11. A method of controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a performance surface, comprising the steps of:

providing a performance surface comprising an array of pressure sensors, each of which generates an output signal representative of an amount of applied pressure when activated;

programming parameter data for reference gestures;

storing the parameter data in memory;

detecting a pattern of activated sensors when a player contacts the array;

comparing a detected pattern of activated sensors with the parameter data stored in memory;

determining a best fit of the detected pattern with the parameter data stored in memory; and

controlling an electronic tone generator in accordance with a determined best fit.

12. A control system for controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a performance surface, the control system comprising:

a performance surface comprising an array of pressure sensors, each generating an output signal representative of an amount of applied pressure when activated;

detecting means for detecting a pattern of sensor output;

a memory capable of storing parameter data for various reference gestures;

comparing means for comparing a detected pattern of sensor output with parameter data stored in the memory;

determining means for determining a best fit of the detected pattern with the parameter data stored in the memory; and

control means for controlling an electronic tone generator in accordance with a determined best fit.

13. A control system for controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a pressure sensitive surface, the control system comprising:

a pressure sensitive surface comprising an array of sensors, each of which provides an output;

detecting means for detecting a surface contact pattern from the sensor output, the surface contact pattern comprising the part of the surface to which at least a threshold amount of pressure is applied;

a memory capable of storing parameter data representative of plural types of reference surface contact patterns;

comparing means for comparing a detected surface contact pattern associated with an incoming gesture with the parameter data stored in the memory, an incoming gesture being a gesture presently contacting the pressure sensitive surface;

determining means for determining which of the reference surface contact patterns best fits the detected surface contact pattern; and

control means for controlling an electronic tone generator in accordance with a determined best fit.

14. The control system of claim 13, wherein the control means controls the electronic tone generator to generate a tone having at least one tone characteristic determined in accordance with at least one corresponding parameter of a particular gesture type when the particular gesture type is determined to have been used by the player, whereby the player may achieve a timbral variation simply by modifying the corresponding parameter of the particular gesture type.

15. The control system of claim 14, wherein the tone characteristic is one of volume, pitch, modulation, and harmonic content.

16. The control system of claim 14, wherein the parameter is one of average pressure value for the particular gesture type and distance from the center of the performance surface to the center of the gesture.

17. A control system for controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a performance surface, the control system comprising:

a performance surface comprising an array of pressure sensors, each of which generates an output signal when activated; and

a computer comprising a memory storing preset data for various reference gestures, wherein the computer scans the array, each scan of the array constituting a frame, and within each frame detects for each sensor whether or not the sensor has been activated, wherein data is collected for each frame and plural frames of data are collected for each incoming gesture;

wherein the computer compares data collected within frames associated with an incoming gesture with parameter data stored in the memory and determines which one of the reference gestures corresponds most closely to the incoming gesture and outputs a control signal corresponding to the reference gesture providing the best fit;

wherein the control signal may be used to control an electronic tone generator.

18. A control system as in claim 17, further comprising a display which shows which of said sensors are activated by a gesture.

19. A control system as in claim 18, wherein said display comprises an array of light emitting diodes, each diode corresponding to a particular sensor, and each diode illuminating when the corresponding sensor is activated by a gesture.

20. The control system of claim 17, wherein the performance surface is circular and is divided into at least first and second concentric circular sections, the first circular section being situated at the center of the performance surface and the second circular section being situated at the edge of the performance surface, wherein the computer stores tone data in memory corresponding to the center portion of a natural musical instrument for the first circular section and tone data corresponding to the edge portion of a natural musical instrument for the second circular section.

21. A control system for controlling an electronic tone generator in accordance with a type of gesture used by a player to contact a performance surface, the control system comprising:

a performance surface comprising an array of sensors, each of which generates an output signal when activated;

a selector by which a player may select at least one of a program mode and a play mode, wherein in the program mode the control system establishes parameter data for reference gestures input by a player and in the play mode the control system determines which one of the reference gestures most closely corresponds to an incoming gesture, an incoming gesture being a gesture presently contacting the performance surface;

a computer which scans the array, each scan of the array constituting a frame, and within each frame detects for each sensor whether or not the sensor has been activated, wherein data is collected for each frame and plural frames of data are collected for each incoming gesture in both of the program and play modes, and further wherein a predetermined number of frames defines a window and data for an incoming gesture is captured within a corresponding window;

wherein the computer comprises a memory for storing collected data, and in the program mode, the computer establishes parameter data for each type of reference gesture in accordance with windows of data collected for the reference gesture, and in the play mode, the computer compares data collected within a window associated with an incoming gesture with parameter data established for the reference gestures and determines which one of the reference gestures corresponds most closely to the incoming gesture and outputs a control signal corresponding to the reference gesture providing the best fit;

wherein the control signal may be used to control an electronic tone generator.

22. A control system according to claim 21, wherein the sensors are arranged such that each sensor has a corresponding position represented by coordinates x,y and wherein the frame data collected by the computer comprises coordinates x,y associated with each activated sensor and an amount of pressure applied to each activated sensor.

23. A control system according to claim 21, wherein the computer calculates k parameters Pi (i=1 to k) for an incoming gesture in accordance with collected frame data and calculates the means for each parameter for each reference gesture in accordance with the equation:

##EQU8## and the standard deviation for each parameter in accordance with the equation;

##EQU9## and wherein the computing means determines which of the reference gestures best fits the incoming gesture by calculating the overall error term for each reference gesture in accordance with the equation;

##EQU10## where E_g =overall error term for a reference g

Pi_inc =ith parameter of an incoming gesture

mean Pi_g =mean of the ith parameter of the reference gesture g

sd Pi_g =standard deviation of the ith parameter of the reference gesture g;

wherein the lowest overall error term corresponds to the reference gesture which best fits the incoming gesture.