Controlling and enhancing electronic musical instruments with video

• US 8,519,250 B2
• Filed: 10/10/2003
• Issued: 08/27/2013
• Est. Priority Date: 05/15/1998
• Status: Expired due to Fees

• Current Assignee: Advance Touchscreen And Gesture Technologies, LLC
• Original Assignee: Lester F. Ludwig
• Inventors: Ludwig, Lester F.
• Primary Examiner(s): FLETCHER, MARLON T
• Application Number:
  US10/683,915
• Publication Number:
  US 20040118268A1
• Time in Patent Office: 3,609 Days
• Field of Search: None
• US Class Current: 84/645
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G10H 1/00 Details of electrophonic musical instruments
• CPC Class Codes: G10H 1/0058 Transmission between separate instruments or between individual components of a musical system G10H1/0083 takes precedence
• CPC Class Codes: G10H 1/0066 using a MIDI interface
• CPC Class Codes: G10H 1/348 Switches actuated by parts of the body other than fingers
• CPC Class Codes: G10H 1/46 Volume control
• CPC Class Codes: G10H 2220/521 Hall effect transducers or similar magnetic field sensing semiconductor devices, e.g. for string vibration sensing or key movement sensing
• CPC Class Codes: G10H 2220/525 Piezoelectric transducers for vibration sensing or vibration excitation in the audio range; Piezoelectric strain sensing, e.g. as key velocity sensor; Piezoelectric actuators, e.g. key actuation in response to a control voltage
• CPC Class Codes: G10H 2230/095 Spint zither, i.e. mimicking any neckless stringed instrument in which the strings do not extend beyond the sounding board
• CPC Class Codes: G10H 2230/101 Spint koto, i.e. mimicking any traditional asian-style plucked zither with movable bridges
• CPC Class Codes: G10H 2230/115 Spint sitar, i.e. mimicking any long-necked plucked string instrument with a large number of additional non-playable sympathetic resonating strings or an additional gourd-like resonating chamber
• CPC Class Codes: G10H 2230/145 Spint guitar keyboard, i.e. mimicking a combination of a guitar-like instrument, with or without strings, and a piano-like keyboard, e.g. with white and black keys arranged like on a piano
• CPC Class Codes: G10H 2230/271 Spint gong, i.e. mimicking circular flat, nippled or bowl-shaped metallic percussion instruments G10H2230/321 takes precedence
• CPC Class Codes: G10H 3/18 using a string, e.g. electric guitar mechanical features G10D1/085
• CPC Class Codes: G10H 3/26 : using electric feedback
• CPC Class Codes: Y10S 84/10 : Feedback

Abstract

The invention provides for the integration of video into electronic music technology. Cameras on instruments, such as guitars, are provided for display during performance, recording, and for creating control signals. Extraction of control signals from camera images, including patterns and gestures in 2D and 3D, may be included. All such outgoing control signals may be MIDI, and extraction algorithms may be selected and controlled by MIDI. A range of video signal generation, video signal processing, and camera image capture control functions relevant to the use of video in live performances may be controlled by MIDI to unify a common control infrastructure for a live performance environment.

Claims

1. A system for extracting and generating control signals from at least one video signal, said system comprising:

a first image processing element for receiving a first incoming video signal, wherein said first image processing element detects at least one image-related component within an image represented by said first incoming video signal; and

a processor for processing said at least one image-related component to generate at least one outgoing control signal comprising at least one of a MIDI continuous controller message and a MIDI note message, the outgoing control signal being generated responsive to content in the image represented by said first incoming video signal.

2. The system according to claim 1, wherein said first incoming video signal originates from a video camera.

3. The system according to claim 2, wherein said video camera is attached to a musical instrument.

4. The system according to claim 2, wherein said incoming video signal is communicated to a video display visible to an audience.

5. The system according to claim 1, wherein an incoming control signal is used by said processor to generate said at least one outgoing control signal.

6. The system according to claim 5, wherein said incoming control signal comprises signals of MIDI format.

7. The system according to claim 1, wherein said processor performs the processing response to a control signal extraction algorithm selected from a plurality of pre-stored control signal extraction algorithms.

8. The system according to claim 1, wherein said processor is configured to recognize at least one pattern in said incoming video signal, wherein said at least one pattern is used to generate said at least one outgoing control signal.

9. The system according to claim 8, wherein said processor is further configured to calculate at least one pattern parameter associated with said at least one pattern in said incoming video signal, wherein said pattern parameter is used to generate said at least one outgoing control signal.

10. The system according to claim 8, wherein said processor is further configured to recognize at least one gesture in said incoming video signal, wherein said recognition is used to generate said at least one outgoing control signal.

11. The system according to claim 10, wherein said processor is further configured to calculate at least one gesture parameter associated with said at least one gesture recognized in said video input signal, wherein said gesture parameter is used to generate said at least one outgoing control signal.

12. The system according to claim 1, said system further comprising:

a second video signal image processing element for receiving a second incoming video signal, wherein said second video signal image processing element detects at least one image-related component from said second incoming video signal, wherein

said processor is further configured to processes said at least one image-related component of said second incoming video signal, wherein said processing of said at least one image-related component of said first incoming video signal and said at least one image-related component of said second incoming video signal results in said at least one outgoing control signal.

13. The system according to claim 12, wherein said second incoming video signal originates from a video camera.

14. The system according to claim 1, said system further comprising:

a second video signal image processing element for receiving a second incoming video signal, wherein said second video signal image processing element detects at least one image-related component from said second incoming video signal, wherein

said processor is further configured to processes said at least one image-related component of said second incoming video signal resulting in a secondary at least one outgoing control signal comprising at least one of a MIDI continuous controller message and a MIDI note message.

15. The system according to claim 12, wherein said processor is further configured to recognize at least one pattern in said second incoming video signal, wherein said at least one pattern is used to generate said at least one outgoing control signal.

16. The system according to claim 15, wherein said processor is further configured to calculate at least one pattern parameter associated with said at least one pattern in said second incoming video signal, wherein said pattern parameter is used to generate said at least one outgoing control signal.

17. The system according to claim 15, wherein said processor is further configured to recognize at least one gesture in said second incoming video signal to generate said at least one outgoing control signal.

18. The system according to claim 17, wherein said processor is further configured to calculate at least one gesture parameter associated with said at least one gesture recognized in said video input signal, wherein said gesture parameter is used to generate said at least one outgoing control signal.

19. The system according to claim 1, wherein said first incoming video signal is processed by a real-time image processor to generate a modified video signal, wherein said real-time image processor is controlled by an incoming control signal.

20. A method for extracting and generating control signals from at least one video signal, said method comprising:

receiving a first incoming video signal at a first image processing element, wherein said first image processing element detects posture of a human hand in an image represented by said first incoming video signal; and

processing the detected posture in the image, resulting in an outgoing control signal comprising at least one of a of MIDI format continuous controller message and a MIDI note message, the outgoing control signal being generated responsive to the detected posture of the human hand.

21. The method according to claim 20, wherein said first incoming video signal originates from a video camera.

22. The method according to claim 21, wherein said video camera is attached to a musical instrument.

23. The method according to claim 21, wherein said incoming video signal is communicated to a video display visible to an audience.

24. The method according to claim 20, wherein an incoming control signal is used by said processor to generate said at least one outgoing control signal.

25. The method according to claim 24, wherein said incoming control signal comprises signals of MIDI format.

26. The method according to claim 20, wherein said processor performs the processing response to a control signal extraction algorithm selected from a plurality of pre-stored control signal extraction algorithms.

27. The method according to claim 20, wherein said processor is configured to recognize at least one pattern in said incoming video signal, wherein said at least one pattern is used to generate said at least one outgoing control signal.

28. The method according to claim 27, wherein said processor is further configured to calculate at least one pattern parameter associated with said at least one pattern in said incoming video signal, wherein said pattern parameter is used to generate said at least one outgoing control signal.

29. The method according to claim 27, wherein said processor is further configured to recognize at least one gesture in said incoming video signal, wherein said recognition is used to generate said at least one outgoing control signal.

30. The method according to claim 29, wherein said processor is further configured to calculate at least one gesture parameter associated with said at least one gesture recognized in said video input signal, wherein said gesture parameter is used to generate said at least one outgoing control signal.

31. The method according to claim 20, said method further comprising:

receiving a second incoming video signal at a second video signal image processing element, wherein said second video signal decomposition element detects at least one image-related component from said second incoming video signal; and

processing said at least one image-related component of said second incoming video signal, wherein said processing of said at least one image-related component of said first incoming video signal and said at least one image-related component of said second incoming video signal results in said at least one outgoing control signal.

32. The method according to claim 31, wherein said second incoming video signal originates from a video camera.

33. The method according to claim 20, said method further comprising:

receiving a second incoming video signal at a second video signal image processing element, wherein said second video signal image processing element detects at least one image-related component from said second incoming video signal; and

processing said at least one image-related component of said second incoming video signal resulting in a secondary at least one outgoing control signal comprising at least one of a MIDI continuous controller message and a MIDI note message.

34. The method according to claim 31, wherein said processor is further configured to recognize at least one pattern in said second incoming video signal, wherein said at least one pattern is used to generate said at least one outgoing control signal.

35. The method according to claim 34, wherein said processor is further configured to calculate at least one pattern parameter associated with said at least one pattern in said second incoming video signal, wherein said pattern parameter is used to generate said at least one outgoing control signal.

36. The method according to claim 34, wherein said processor is further configured to calculate at least one gesture in said second incoming video signal to generate said at least one outgoing control signal.

37. The method according to claim 36, wherein said processor is further configured to calculate at least one gesture parameter associated with said at least one gesture recognized in said video input signal, wherein said gesture parameter is used to generate said at least one outgoing control signal.

38. The method according to claim 20, wherein said first incoming video signal is processed by a real-time image processor to generate a modified video signal, wherein said real-time image processor is controlled by an incoming control signal.