Tactile, visual, and array controllers for real-time control of music signal processing, mixing, video, and lighting

• US 20020005108A1
• Filed: 03/19/2001
• Published: 01/17/2002
• Est. Priority Date: 05/15/1998
• Status: Active Grant

• Current Assignee: Advance Touchscreen And Gesture Technologies, LLC
• Original Assignee: Lester Frank Ludwig
• Inventors: Ludwig, Lester Frank
• Granted Patent: US 6,570,078 B2
• Time in Patent Office: Days
• US Class Current: 84/600
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G10H 1/00 Details of electrophonic musical instruments
• CPC Class Codes: G10H 1/0058 Transmission between separate instruments or between individual components of a musical system G10H1/0083 takes precedence
• CPC Class Codes: G10H 1/0066 using a MIDI interface
• CPC Class Codes: G10H 1/348 Switches actuated by parts of the body other than fingers
• CPC Class Codes: G10H 1/46 Volume control
• CPC Class Codes: G10H 2220/521 Hall effect transducers or similar magnetic field sensing semiconductor devices, e.g. for string vibration sensing or key movement sensing
• CPC Class Codes: G10H 2220/525 Piezoelectric transducers for vibration sensing or vibration excitation in the audio range; Piezoelectric strain sensing, e.g. as key velocity sensor; Piezoelectric actuators, e.g. key actuation in response to a control voltage
• CPC Class Codes: G10H 2230/095 Spint zither, i.e. mimicking any neckless stringed instrument in which the strings do not extend beyond the sounding board
• CPC Class Codes: G10H 2230/101 Spint koto, i.e. mimicking any traditional asian-style plucked zither with movable bridges
• CPC Class Codes: G10H 2230/115 Spint sitar, i.e. mimicking any long-necked plucked string instrument with a large number of additional non-playable sympathetic resonating strings or an additional gourd-like resonating chamber
• CPC Class Codes: G10H 2230/145 Spint guitar keyboard, i.e. mimicking a combination of a guitar-like instrument, with or without strings, and a piano-like keyboard, e.g. with white and black keys arranged like on a piano
• CPC Class Codes: G10H 2230/271 Spint gong, i.e. mimicking circular flat, nippled or bowl-shaped metallic percussion instruments G10H2230/321 takes precedence
• CPC Class Codes: G10H 3/18 using a string, e.g. electric guitar mechanical features G10D1/085
• CPC Class Codes: G10H 3/26 : using electric feedback
• CPC Class Codes: Y10S 84/10 : Feedback

Abstract

A system and method for real-time controlling of signal processors, synthesizers, musical instruments, MIDI processors, lighting, video, and special effects in performance, recording, and composition environments using images derived from tactile sensors, pressure sensor arrays, optical transducer arrays, chemical sensor arrays, body sensor arrays, and numerical computation. The invention provides for null/contact touch-pads, pressure sensor arrays, and body sensor arrays as tactile control interfaces, for video cameras and light sensor arrays as optical transducers, for chemical sensor arrays, and for other numerical image generation from computer processing or numeric simulation. Tactile transducers may be put on instrument keys of conventional instruments, be attached to existing instruments, or be used to create entirely new instrument or controller configurations. Chemical sensor arrays and for other numerical image generation from computer processing or numeric simulation can be used to monitor or simulate natural physical phenomenon such as self-organizing process behavior or environmental conditions. Arrays of scalar or vector values are processed to extract pattern boundaries, geometric properties of pixels within the pattern boundaries (geometric center, weighted moments, etc.), and higher-level derived information (senses of rotation, segmented regions, pattern classification, syntaxt, grammers, sequences, etc.) which are used to create control signals to external audio, visual, and control equipment or algorithms. The invention also provides for MIDI and non-MIDI control signals.

Claims

1. A system for the generation of control signals used to control audio, visual and control systems based on the processing of image and sensor array data, the system comprising:

a source of at least one data array comprising pixel data comprising scalar-valued or vector-valued data;

at least one processor for processing said data array;

an algorithm and/or circuitry for extracting information comprising at least one of

pattern boundaries;

pattern geometric coordinates;

geometric weighted moments defined by the pattern and the pixel data in combination;

pattern classifications,

wherein the extracted information is used to create control signals.

3. The system of claim 1 wherein the source of said data array is a pressure sensor array.

4. The system of claim 3 wherein said pressure sensor array is attached to the key of a musical instument keyboard.

5. The system of claim 1 wherein the source of said data array is a video camera.

6. The system of claim 1 wherein the source of said data array is any one of:

a chemical sensor array;

an enviromental sensor array;

or

a body sensor array.

7. The system of claim 1 wherein the process for generation of control signals from the extracted information is variably defined.

8. The system of claim 7 wherein a plurality of variably defined generation processes may be stored and recalled.

2. A method for the generation of control signals used to control audio, visual and control systems based on the processing of image and sensor array data, the system utilizing:

a source of at least one data array comprising pixel data utilizing scalar-valued or vector-valued data;

at least one processor for processing said data array;

an algorithm and/or circuitry utilized to extract information comprising at least one of;

pattern boundaries;

pattern geometric coordinates;

geometric weighted moments defined by the pattern and the pixel data in combination;

pattern classifications,

wherein the extracted information is used to create control signals.

9. The method of claim 2 wherein the process for generation of control signals from the extracted information is variably defined.

10. The method of claim 10 wherein a plurality of variably defined generation processes may be stored and recalled.