Computer interface for polyphonic stringed instruments

• US 8,581,086 B2
• Filed: 06/21/2011
• Issued: 11/12/2013
• Est. Priority Date: 07/10/2008
• Status: Expired due to Fees

• Current Assignee: Kesumo, LLC
• Original Assignee: Kesumo, LLC
• Inventors: McMillen, Keith, Shaver, Chris
• Primary Examiner(s): Qin, Jianchun
• Application Number:
  US13/165,587
• Publication Number:
  US 20110303075A1
• Time in Patent Office: 875 Days
• Field of Search: 846/45
• US Class Current: 84/645
• CPC Class Codes: G10H 1/125 using a digital filter
• CPC Class Codes: G10H 1/342 for guitar-like instruments with or without strings and with a neck on which switches or string-fret contacts are used to detect the notes being played electric guitars in which the tones are generated by the vibration of strings G10H3/18
• CPC Class Codes: G10H 2210/031 Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
• CPC Class Codes: G10H 2210/155 Musical effects
• CPC Class Codes: G10H 2210/311 Distortion, i.e. desired non-linear audio processing to change the tone colour, e.g. by adding harmonics or deliberately distorting the amplitude of an audio waveform
• CPC Class Codes: G10H 2220/155 User input interfaces for electrophonic musical instruments
• CPC Class Codes: G10H 2220/161 with 2D or x/y surface coordinates sensing
• CPC Class Codes: G10H 2220/361 Mouth control in general, i.e. breath, mouth, teeth, tongue or lip-controlled input devices or sensors detecting, e.g. lip position, lip vibration, air pressure, air velocity, air flow or air jet angle
• CPC Class Codes: G10H 2220/395 Acceleration sensing or accelerometer use, e.g. 3D movement computation by integration of accelerometer data, angle sensing with respect to the vertical, i.e. gravity sensing
• CPC Class Codes: G10H 2240/285 USB, i.e. either using a USB plug as power supply or using the USB protocol to exchange data
• CPC Class Codes: G10H 2240/311 MIDI transmission G10H2240/056 takes precedence
• CPC Class Codes: G10H 2250/311 Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation
• CPC Class Codes: G10H 2250/501 Formant frequency shifting, sliding formants
• CPC Class Codes: G10H 2250/615 Waveform editing, i.e. setting or modifying parameters for waveform synthesis
• CPC Class Codes: G10H 3/186 Means for processing the signal picked up from the strings filtering G10H1/12
• CPC Class Codes: G10H 3/188 for converting the signal to digital format transmission using a MIDI interface G10H1/0066

Abstract

An interface device is described that allows the audio signals from a polyphonic stringed instrument to be introduced into a personal computer environment for feature extraction and signal processing.

Claims

1. A computer-implemented method for processing audio signals for a stringed instrument, comprising:

receiving a serial data stream with a serial data interface of a computing device, the serial data stream encoding a plurality of digital audio signals, each digital audio signal representing one of a plurality of strings of the stringed instrument;

extracting the encoded digital audio signals from the serial data stream using the computing device;

processing each of the extracted digital audio signals with the computing device to detect string events for one or more of the plurality of strings of the stringed instrument, including conducting a time-domain analysis of each of the extracted digital audio signals to identify local maxima and minima associated with each of the extracted digital audio signals in the time domain, and identifying the string events with reference to the local maxima and minima; and

classifying each of the string events as corresponding to one of a plurality of performance techniques used by a musician to generate the corresponding string event, the plurality of performance techniques including picking, plucking, tapping, hammering on, and hammering off.

2. The method of claim 1 further comprising processing the extracted digital audio signals including one or more of dynamics processing, equalization, pitch-shifting, filtering, frequency modulation, amplitude modulation, delay, reverberation, distortion, wave shaping, driving wave tables, stimulating resonances, gating, or limiting.

3. The method of claim 1 further comprising processing each of the extracted digital audio signals using a plurality of processing modules that simultaneously process each extracted digital audio signal in both a time domain and a frequency domain.

4. The method of claim 1 further comprising recording each of the digital audio signals for subsequent processing.

5. The method of claim 1 further comprising generating graphical representations corresponding to each of the digital audio signals, the graphical representations representing one or more of pitch, dynamics, or timbre for the corresponding string of the string instrument.

6. The method of claim 1 further comprising controlling a special effect using information extracted from the serial digital stream.

7. The method of claim 1 further comprising identifying a beginning of a note with reference to one or more of the string events.

8. The method of claim 1 wherein classifying the string events is done using a neural network.

9. The method of claim 1 further comprising extracting a plurality of audio signal characteristics from the extracted digital audio signals, wherein processing of the extracted digital audio signals is done with reference to the audio signal characteristics.

10. The method of claim 9 wherein the audio signal characteristics include any of a continuous pitch of a fundamental harmonic of a string, an amplitude, a centroid, brightness, even/odd harmonic balance, noise, spectral shape, or complete spectrum.

11. The method of claim 1 further comprising generating a plurality of acoustic instrument messages with reference to the extracted digital audio signals, each acoustic instrument message summarizing spectral information corresponding to a particular one of the plurality of strings of the stringed instrument.

12. The method of claim 1 wherein conducting the time-domain analysis comprises identifying positive-polarity segments and negative-polarity segments of each extracted digital audio signal, classifying the positive-polarity segments into positive-polarity event candidates and negative-polarity segments into negative-polarity event candidates for each extracted digital audio signal, and classifying at least some of the positive-polarity and negative polarity event candidates for one or more of the extracted digital audio signals as the string events.

13. The method of claim 1 wherein processing each of the extracted digital audio signals comprises tracking a fundamental harmonic and one or more overtones independently.

14. The method of claim 13 further comprising determining a pitch of the fundamental harmonic.

15. The method of claim 14 wherein the pitch of the fundamental harmonic is determined with reference to fret scanning information.

16. The method of claim 13 further comprising determining an even/odd harmonic balance from the fundamental harmonic and the one or more overtones.

17. A computer program product for processing audio signals for a stringed instrument, the computer program product comprising at least one non-transitory computer-readable storage medium having computer program instructions stored therein configured to enable at least one computing device to:

receive a serial data stream with a serial data interface of a computing device, the serial data stream encoding a plurality of digital audio signals, each digital audio signal representing one of a plurality of strings of the stringed instrument;

extract the encoded digital audio signals from the serial data stream using the computing device;

process each of the extracted digital audio signals with the computing device to detect string events for one or more of the plurality of strings of the stringed instrument, including conducting a time-domain analysis of each of the extracted digital audio signals to identify local maxima and minima associated with each of the extracted digital audio signals in the time domain, and identifying the string events with reference to the local maxima and minima; and

classify each of the string events as corresponding to one of a plurality of performance techniques used by a musician to generate the corresponding string event, the plurality of performance techniques including picking, plucking, tapping, hammering on, and hammering off.

18. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to process the extracted digital audio signals using dynamics processing, equalization, pitch-shifting, filtering, frequency modulation, amplitude modulation, delay, reverberation, distortion, wave shaping, driving wave tables, stimulating resonances, gating, or limiting.

19. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to process each of the extracted digital audio signals using a plurality of processing modules that simultaneously process each extracted digital audio signal in both a time domain and a frequency domain.

20. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to record each of the digital audio signals for subsequent processing.

21. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to generate graphical representations corresponding to each of the digital audio signals, the graphical representations representing one or more of pitch, dynamics, or timbre for the corresponding string of the string instrument.

22. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to control a special effect using information extracted from the serial digital stream.

23. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to identify a beginning of a note with reference to one or more of the string events.

24. The computer program product of claim 17 wherein the computer program instructions are configured to enable the at least one computing device to classify the string events using a neural network.

25. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to extract a plurality of audio signal characteristics from the extracted digital audio signals, and process the extracted digital audio signals with reference to the audio signal characteristics.

26. The computer program product of claim 25 wherein the audio signal characteristics include any of a continuous pitch of a fundamental harmonic of a string, an amplitude, a centroid, brightness, even/odd harmonic balance, noise, spectral shape, or complete spectrum.

27. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to generate a plurality of acoustic instrument messages with reference to the extracted digital audio signals, each acoustic instrument message summarizing spectral information corresponding to a particular one of the plurality of strings of the stringed instrument.

28. The computer program product of claim 17 wherein the computer program instructions are configured to enable the at least one computing device to conduct the time-domain analysis by identifying positive-polarity segments and negative-polarity segments of each extracted digital audio signal, classifying the positive-polarity segments into positive-polarity event candidates and negative-polarity segments into negative-polarity event candidates for each extracted digital audio signal, and classifying at least some of the positive-polarity and negative polarity event candidates for one or more of the extracted digital audio signals as the string events.

29. The computer program product of claim 17 wherein the computer program instructions are configured to enable the at least one computing device to process each of the extracted digital audio signals by tracking a fundamental harmonic and one or more overtones independently.

30. The computer program product of claim 29 wherein the computer program instructions are further configured to enable the at least one computing device to determine a pitch of the fundamental harmonic.

31. The computer program product of claim 30 wherein the computer program instructions are configured to enable the at least one computing device to determine the pitch of the fundamental harmonic with reference to fret scanning information.

32. The computer program product of claim 29 wherein the computer program instructions are further configured to enable the at least one computing device to determine an even/odd harmonic balance from the fundamental harmonic and the one or more overtones.