Method and device for detecting and separating voices in a complex musical composition

• US 5,210,366 A
• Filed: 06/10/1991
• Issued: 05/11/1993
• Est. Priority Date: 06/10/1991
• Status: Expired due to Fees

• Current Assignee: Richard O. Sykes Jr
• Original Assignee: Richard O. Sykes Jr
• Inventors: Sykes, Richard O. Jr.
• Primary Examiner(s): Shoop, Jr., William M.
• Assistant Examiner(s): KIM, HELEN
• Application Number:
  US07/712,516
• Time in Patent Office: 701 Days
• Field of Search: 84/627, 84/663, 84/625, 84/604, 84/622, 84/632, 84/615, 84/616, 84/DIG. 11, 84/454, 84/462, 84/672, 84/673, 84/675, 84/681, 381/43, 381/46, 381/47, 340/566
• US Class Current: 84/616
• CPC Class Codes: G10H 1/0033 Recording/reproducing or transmission of music for electrophonic musical instruments
• CPC Class Codes: G10H 1/12 by filtering complex waveforms G10H1/14, G10H1/16 take precedence
• CPC Class Codes: G10H 2210/056 for extraction or identification of individual instrumental parts, e.g. melody, chords, bass; Identification or separation of instrumental parts by their characteristic voices or timbres
• CPC Class Codes: G10H 2210/086 for transcription of raw audio or music data to a displayed or printed staff representation or to displayable MIDI-like note-oriented data, e.g. in pianoroll format
• CPC Class Codes: G10H 2250/031 Spectrum envelope processing
• CPC Class Codes: G10H 3/125 Extracting or recognising the pitch or fundamental frequency of the picked up signal
• CPC Class Codes: Y10S 84/11 Frequency dividers

Abstract

A system and method for detecting, separating and recording the individual voices in a musical composition performed by a plurality of instruments. The electrical waveform signal for the multi-voiced musical composition is fed to a waveform signal converter to convert the waveform signal to a frequency spectrum representation. The frequency spectrum representation is fed to a frequency spectrum comparator where it is compared to predetermined steady-state frequency spectrum representations for a particular musical instrument. Upon detecting the presence of a frequency spectrum representation corresponding to a predetermined steady-state frequency spectrum representation, the detected frequency spectrum representation and measured growth and decay frequency spectrum representations are fed to a waveform envelope comparator and compared to predetermined waveform envelopes, i.e. frequency spectrum representations during the growth, steady-state and decay periods of the waveform signal. Upon detecting the presence of a waveform envelope corresponding to a predetermined waveform envelope, the steady-state and transient properties of the detected frequency spectrum representation are recorded and converted to an electrical waveform signal for output as music data for an individual voice.

Claims

1. A sound signal analyzer for automatic detection and separation of a single voice in a complex musical composition comprising

(a) waveform signal conversion means responsive to an electrical waveform signal having a frequency corresponding to the frequency of an audible sound wave for the complex musical composition for converting the electrical waveform signal to a complex frequency spectrum representation for the complex musical composition;

(b) frequency spectrum representation comparison means responsive to the complex frequency spectrum representation derived by said conversion means for comparing the complex frequency spectrum representation to predetermined steady-state, single-voice frequency spectrum representations corresponding to notes capable of being produced by a single instrument included in the complex musical composition;

(c) single-voice frequency spectrum representation detection means responsive to said frequency spectrum representation comparison means for detecting the presence of a predetermined steady-state, single-voice frequency spectrum representation corresponding to a note capable of being produced by the single instrument; and

(d) frequency spectrum representation separation means responsive to the predetermined steady-state, single-voice frequency spectrum representation detected by the single-voice frequency spectrum representation detection means for separating the detected steady-state, single-voice frequency spectrum representation and respective complex frequency spectrum representations in growth and decay periods of the note corresponding to the detected frequency spectrum representation, said steady-state, growth and decay frequency spectrum representations, in combination, defining a measured waveform envelope representation.

2. A sound signal analyzer as in claim 1 further including

(e) waveform envelope representation comparison means responsive to the measured waveform envelope representation for comparing the measured waveform envelope representation to predetermined single-voice waveform envelope representations corresponding to notes capable of being produced by the single instrument;

(f) waveform envelope representation detection means responsive to said waveform envelope representation comparison means for detecting the presence of a predetermined single-voice waveform envelope representation corresponding to the note depicted by the steady-state, single-voice frequency spectrum representation included in the measured waveform envelope representation; and

(g) waveform envelope representation separation means responsive to the waveform envelope representation detected by the waveform envelope representation detection means for separating the detected single-voice waveform envelope representation.

3. A sound signal analyzer as in claim 2 further including data output means to output the detected single-voice waveform envelope representation as music data.

4. A sound signal analyzer as in claim 3 wherein said music data comprises audible musical sound.

5. A sound signal analyzer as in claim 3 wherein said music data comprises music notation.

6. A sound signal analyzer as in claim 3 wherein said data output means comprises, in combination, means to record the detected single-voice waveform envelope representation in a readable form and means to convert the recorded waveform envelope representation to an electrical waveform signal.

7. A sound signal analyzer as in claim 2 comprising at least two frequency spectrum representation comparison means, at least two single-voice frequency spectrum representation detection means, at least two frequency spectrum representation separation means, at least two waveform envelope representation detection means, and at least two waveform envelope representation separation means, corresponding to at least two distinct instruments in the complex musical composition.

8. A sound signal analyzer as in claim 7 further including at least two data output to output the respective detected single-voice waveform envelope representations as music data corresponding to at least two distinct instruments in the complex musical composition.

9. A sound signal analyzer as in claim 8 further including at least two filtering means disposed between respective waveform envelope representation separation means and successive frequency spectrum representation comparison means.

11. A sound signal analyzer as in claim 1 comprising at least two frequency spectrum representation comparison means, at least two single-voice frequency spectrum representation detection means, and at least two frequency spectrum representation separation means, corresponding to at least two distinct instruments in the complex musical composition.

10. A sound signal analyzer as in claim 10 further including clock means for sequentially cuing the complex frequency spectrum representations derived by the waveform signal conversion means and filtered by said filtering means.

12. A voice detection and separation system for detecting and separating a single voice in a complex musical composition comprising

(i) sound wave conversion means responsive to audible sound waves of the complex musical composition for converting the sound waves into an electrical waveform signal;

(ii) amplifier means for amplifying the electrical waveform signal derived by said sound wave conversion means; and

(iii) a sound signal analyzer comprising;

(a) waveform signal conversion means responsive to the amplified electrical waveform signal for converting the electrical waveform signal to a complex frequency spectrum representation for the complex musical composition,

(b) frequency spectrum representation comparison means responsive to the complex frequency spectrum representation derived by said waveform signal conversion means for comparing the complex frequency spectrum representation to predetermined steady-state, single-voice frequency spectrum representations corresponding to notes capable of being produced by a single instrument included in the complex musical composition,

(c) single-voice frequency spectrum representation detection means responsive to said frequency spectrum representation comparison means for detecting the presence of a predetermined steady-state, single-voice frequency spectrum representation corresponding to a note capable of being produced by the single instrument, and

(d) frequency spectrum representation separation means responsive to the predetermined steady-state, single-voice frequency spectrum representation detected by the single-voice frequency spectrum representation detection means for separating the detected steady-state, single-voice frequency spectrum representation and respective complex frequency spectrum representations in growth and decay periods of the note corresponding to the detected frequency spectrum representation, said steady-state, growth and decay frequency spectrum representations, in combination, defining a measured waveform envelope representation.

13. A voice detection and separation system as in claim 12 wherein said audible sound waves are produced by a live performance of a plurality of musical instruments.

14. A voice detection and separation system as in claim 12 wherein said audible sound waves are produced by a stored performance of a plurality of musical instruments.

15. A voice detection and separation system as in claim 12 wherein said waveform signal conversion means comprises means to automatically separate fundamental and overtone frequency components of the complex musical composition and simultaneously measure the frequency and amplitude of the respective fundamental and overtone frequency components.

16. A voice detection and separation system as in claim 15 wherein said frequency spectrum representation comparison means, said single-voice frequency spectrum detection means and said frequency spectrum representation separation means comprise, in combination, a frequency spectrum comparator made operable by an algorithm providing directives to respectively compare the frequency and amplitude of said respective fundamental and overtone frequency components of the complex musical composition to fundamental and overtone frequency components of a single voice, detect the presence of said single-voice frequency components in said complex frequency components, and separate said single-voice frequency components from said complex frequency components.

17. A voice detection and separation system as in claim 15 wherein said frequency spectrum comparator includes an accumulating memory for temporary storage of frequency spectrum representations in growth and decay periods of the note corresponding to the detected frequency spectrum representation derived by said single-voice frequency spectrum representation detection means.

18. A voice detection and separation system for detecting and separating a voice in a complex musical composition comprising

(i) sound wave conversion means responsive to audible sound wave of the complex musical composition for converting the sound waves into an electrical waveform signal;

(ii) amplifier means for amplifying the electrical waveform signal derived by said sound wave conversion means; and

(iii) a sound signal analyzer comprising;

(a) waveform signal conversion means responsive to the amplified electrical waveform signal frequency corresponding to the frequency of an audible sound wave for the complex musical for converting the electrical waveform signal to a complex frequency spectrum representation for the complex musical composition,

(b) frequency spectrum representation comparison means responsive to the complex frequency spectrum representation derived by said conversion means for comparing the complex frequency spectrum representation to predetermined steady-state, single-voice frequency spectrum representations corresponding to notes capable of being produced by a single instrument included in the complex musical composition,

(c) single-voice frequency spectrum representation detection means responsive to said frequency spectrum representation comparison means for detecting the presence of a predetermined steady-state, single-voice frequency spectrum representation corresponding to a note capable of being produced by the single instrument,

(d) frequency spectrum representation separation means responsive to the predetermined steady-state, single-voice frequency spectrum representation detected by the single-voice frequency spectrum representation detection means for separating the detected steady-state, single-voice frequency spectrum representation and respective complex frequency spectrum representations in growth and decay periods of the note corresponding to the detected frequency spectrum representation, said steady-state, growth and decay frequency spectrum representations, in combination, defining a measured waveform envelope representation,

(e) waveform envelope representation comparison means responsive to the measured waveform envelope representation for comparing the measured waveform envelope representation to predetermined single-voice waveform envelope representations corresponding to notes capable of being produced by the single instrument,

(f) waveform envelope representation detection means responsive to said waveform envelope representation comparison means for detecting the presence of a predetermined single-voice waveform envelope representation corresponding to the note depicted by the steady-state, single-voice frequency spectrum representation included in the measured waveform envelope representation, and

(g) waveform envelope representation separation means responsive to the waveform envelope representation detected by the waveform envelope representation detection means for separating the detected single-voice waveform envelope representation.

19. A voice detection and separation system as in claim 18 wherein said audible sound waves are produced by a live performance of at least two musical instruments.

20. A voice detection and separation system as in claim 18 wherein said audible sound waves are produced by a stored performance of at least two musical instruments.

21. A voice detection and separation system as in claim 18 wherein said audible sound waves are produced by a live performance of at least one musical instrument in combination with a stored performance of at least one musical instrument.

22. A voice detection and separation system as in claim 18 wherein said waveform signal conversion means comprises means to automatically separate fundamental and overtone frequency components of the complex musical composition and simultaneously measure the frequency and amplitude of the respective fundamental and overtone frequency components.

23. A voice detection and separation system as in claim 22 wherein said frequency spectrum representation comparison means, said single-voice frequency spectrum detection means and said frequency spectrum representation separation means comprise, in combination, a frequency spectrum comparator made operable by an algorithm providing directives to respectively compare the frequency and amplitude of said respective fundamental and overtone frequency components of the complex musical composition to fundamental and overtone frequency components of a single voice, detect the presence of said single-voice frequency components in said complex frequency components, and separate said single-voice frequency components from said complex frequency components.

24. A voice detection and separation system as in claim 23 wherein said waveform envelope representation comparison means, said waveform envelope representation detection means and said waveform envelope representation separation means, in combination, comprise a waveform envelope comparator made operable by an algorithm providing directives to respectively compare said separated single-voice frequency components to stored transient properties of a single voice, detect the presence of said stored transient properties in said single-voice frequency components, and separate the transient properties and the frequency components of said detected single voice.

25. A voice detection and separation system as in claim 23, said sound signal analyzer further including

(h) a key comparator communicating with said frequency spectrum comparator via a gate controller, said key comparator comprising key memory means for storing musical keys and associated notes, said gate controller comprising means to effect the operation of said frequency spectrum comparator in accordance to the operation of an algorithm in said key comparator providing directives to determine the key of the complex musical composition.

26. A voice detection and separation system as in claim 25 further including means to selectively program said key memory means.

27. A voice detection and separation system as in claim 25 wherein said key comparator includes means to sample the detected frequency spectrum representation in said frequency spectrum comparator and means to compare said sampled frequency spectrum representation to musical keys and associated notes stored in said key memory means.

28. A voice detection and separation system as in claim 25 wherein said gate controller includes means to shift the stored frequency spectrum representations in said frequency spectrum comparator.

29. A voice detection and separation system as in claim 25 wherein said separated single-voice frequency spectrum representation is supplied to an auxiliary device.

30. A voice detection and separation system as in claim 29 wherein said auxiliary device comprises a note transposer for transposing in real time the separated single-voice frequency spectrum representation to a note in said key memory means.

31. A voice detection and separation system as in claim 25 further including means to supply the stored musical keys and associated notes to a display device.

32. A voice detection and separation system as in claim 25 further including means to formulate subsets of said musical keys and associated notes by counting the number of times a specific note within a musical key is sounded within a particular period of time.

33. A voice detection and separation system as in claim 22 wherein said frequency spectrum comparator includes an accumulating memory for temporary storage of frequency spectrum representations in growth and decay periods of the note corresponding to the detected frequency spectrum representation derived by said single-voice frequency spectrum representation detection means.

34. A voice detecting and separation system for detecting and separating a single voice in a complex musical composition comprising

(i) signal generating means for generating a composition electrical waveform signal corresponding to audible sound waves of the complex musical composition,

(ii) voice detecting and separating means connected to said signal generating means for detecting a voice electrical waveform signal corresponding to a tonal structure of an individual voice in the complex musical composition and for separating the detected voice electrical waveform signal from the composition electrical waveform signal,

said voice detecting and separating means detecting the voice electrical waveform signal by comparing the composition electrical waveform signal to predetermined instantaneous and transient properties of tonal structure representations for the individual voice.

35. A voice detection and separation system as in claim 34 wherein said voice detecting and separating means further detects the voice electrical waveform signal by comparing said voice electrical waveform signal properties of tonal structure representations for the individual voice determinable as a function of the key of the complex musical composition.

36. A voice detection and separation system as in claim 34 further including recording means connected to said voice detecting and separating means for recording the detected voice electrical waveform signal.

37. A voice detection and separation system as in claim 36 further including music data output means connected to said recording means for outputting the detected voice electrical waveform signal as music data.

38. A voice detection and separation system as in claim 37 wherein said music data comprises audible sounds.

39. A voice detection and separation system as in claim 37 wherein said music data comprises music notation.

40. A voice detection and separation system as in claim 34 wherein said voice detecting and separating means comprises

a waveform signal converter connected to said signal generating means for converting the electrical waveform signal corresponding to a tonal structure of the complex musical composition into a frequency spectrum representation for the tonal structure of the complex musical composition;

a frequency spectrum comparator connected to said waveform signal converter for detecting and separating a frequency spectrum representation for an individual voice in the complex musical composition by comparing the frequency spectrum representation for the tonal structure of the complex musical composition with a plurality of predetermined instantaneous frequency spectrum representations for the individual voice; and

a waveform envelope comparator connected to said frequency spectrum comparator for detecting and separating a waveform envelope representation for the frequency spectrum representation for the individual voice in the complex musical composition by comparing the waveform envelope representation for the individual voice with a plurality of predetermined waveform envelope representations for the individual voice.

41. A voice detection and separation system as in claim 40 wherein said plurality of predetermined instantaneous frequency spectrum representations comprise a plurality of instantaneous frequency spectrum representations corresponding to notes capable of being produced by the individual voice.

42. A voice detection and separation system as in claim 41 wherein said plurality of predetermined waveform envelope representations comprise a plurality of waveform envelope representations corresponding to the notes capable of being produced by the individual voice.

43. A voice detection and separation system as in claim 34 wherein a plurality of voice detecting and separating means are connected to said signal generating means corresponding to the plurality of voices in the complex musical composition.

44. A sound signal analyzer for detecting and separating individual voices in a composition electrical waveform signal corresponding to a tonal structure of a complex musical composition, said sound signal analyzer comprising

a waveform signal converter connected to the electrical waveform signal for the tonal structure of the complex musical composition comprising means to convert the electrical waveform signal into a frequency spectrum representation for the complex musical composition;

at least one frequency spectrum comparator connected to the waveform signal converter comprising means to detect and separate a frequency spectrum representation for an individual voice in the complex musical composition; and

at least one waveform envelope comparator corresponding in number to the frequency spectrum comparators and respectively connected to the frequency spectrum comparators comprising means to detect and separate a waveform envelope representation for the frequency spectrum representation for the individual voice detected and separated by the respective frequency spectrum comparators.

45. A sound signal analyzer as in claim 44 further including a clock means for sequentially queuing the frequency spectrum representations for the complex musical composition to the respective frequency spectrum comparators.

46. A sound signal analyzer as in claim 45 further including at least one filter means connected between a waveform envelope comparator and the waveform signal converter for extracting the frequency spectrum representation for the individual voice detected and separated by a frequency spectrum comparator from the frequency spectrum representation for the complex musical composition.

47. A method of detecting and separating individual voices in a complex musical composition comprising the steps of

generating an electrical waveform signal corresponding to a tonal structure of the complex musical composition;

converting the electrical waveform signal into a frequency spectrum representation for the tonal structure of the complex musical composition;

comparing the frequency spectrum representation for the tonal structure of the complex musical composition to a plurality of predetermined instantaneous frequency spectrum representations for at least one individual voice;

separating a frequency spectrum representation for an individual voice from the frequency spectrum representation for the complex musical composition;

comparing the transient properties of a waveform envelope of the separated frequency spectrum representation to a plurality of predetermined waveform envelope representations for at least one individual voice; and

separating the frequency spectrum representation and the transient properties of the waveform envelope for the respective individual voices.

48. A method as in claim 47 further including the step of recording the separated frequency spectrum representation and the transient properties of the waveform envelope for the individual voice as music data.

49. A method as in claim 48 further including the step of sequentially queuing the frequency spectrum representation for the complex musical composition prior to comparing the frequency spectrum representation to subsequent plurality of predetermined instantaneous frequency spectrum representation for an individual voice.

50. A method as in claim 49 further including the step of extracting the separated frequency spectrum representation for an individual voice from the frequency spectrum representation for the complex musical composition prior to comparing the frequency spectrum representation to subsequent plurality of predetermined instantaneous frequency spectrum representations for an individual voice.

51. A method for automatically detecting and separating a single voice in a complex musical composition comprising

(a) converting audible sound waves of the complex musical composition to an electrical waveform signal;

(b) converting the electrical waveform signal to a complex frequency spectrum representation;

(c) comparing the complex frequency spectrum representation to predetermined steady-state, single voice frequency spectrum representations corresponding to notes capable of being produced by a single instrument of the complex musical composition;

(d) detecting the presence of a predetermined steady-state, single-voice frequency spectrum representation corresponding to a note capable of being produced by the single instrument; and

(e) separating the detected frequency spectrum representation and associated complex frequency spectrum representations in the respective growth and decay periods of the note corresponding to the detected frequency spectrum representation.

52. A method as in claim 51 further comprising

(f) comparing the detected frequency spectrum representation and associated complex frequency spectrum representations to predetermined waveform envelopes corresponding to notes capable of being produced by the single instrument;

and

(g) detecting the presence in the detected frequency spectrum representation and associated complex frequency spectrum representations of a predetermined waveform envelope corresponding to the detected note.