Musical synthesizer capable of expressive phrasing

• US 6,316,710 B1
• Filed: 09/27/1999
• Issued: 11/13/2001
• Est. Priority Date: 09/27/1999
• Status: Expired due to Fees

• Current Assignee: Eric Lindemann
• Original Assignee: Eric Lindemann
• Inventors: Lindemann, Eric
• Primary Examiner(s): Fletcher, Marlon T.
• Application Number:
  US09/406,459
• Time in Patent Office: 778 Days
• Field of Search: 846/00--06, 846/09--12, 846/22--27, 846/34--36, 846/49--52, 846/59--60, 846/66--68
• US Class Current: 84/609
• CPC Class Codes: G10H 2210/095 Inter-note articulation aspects, e.g. legato or staccato
• CPC Class Codes: G10H 2240/056 MIDI or other note-oriented file format
• CPC Class Codes: G10H 7/02 in which amplitudes at successive sample points of a tone waveform are stored in one or more memories

Abstract

The present invention describes a device and methods for synthesizing a musical audio signal. The invention includes a device for storing a collection of sound segments taken from idiomatic musical performances. Some of these sound segments include transitions between musical notes such as the slur from the end of one note to the beginning of the next. Much of the complexity and expressivity in musical phrasing is associated with the complex behavior of these transition segments. The invention further includes a device for generating a sequence of sound segments in response to an input control sequence—e.g. a MIDI sequence. The sound segments are associated with musical gesture types. The gesture types include attack, release, transition, and sustain. The sound segments are further associated with musical gesture subtypes. Large upward slur, small upward slur, large downward slur, and small downward slur are examples of subtypes of the transition gesture type. Event patterns in the input control sequence lead to the generation of a sequence of musical gesture types and subtypes, which in turn leads to the selection of a sequence of sound segments. The sound segments are combined to form an audio signal and played out by a sound segment player. The sound segment player pitch-shifts and intensity-shifts the sound segments in response to the input control sequence.

Claims

1. A musical synthesizer for synthesizing an ouptut audio signal in response to an input control sequence, comprising:

sound segment storage means for storing a collection of sound segments, wherein said collection includes a plurality of transitions between musical tones;

sound segment sequencer means responsive to said input control sequence for selecting a sequence of sound segments, including segments corresponding to transitions between musical tones, from said sound segment storage means; and

sound segment player means for combining and playing out said sequence of sound segments to form said output audio signal.

2. The apparatus according to claim 1 wherein said input control sequence includes note-on events, and wherein each said note-on event includes a pitch value.

3. The apparatus according to claim 1 wherein said input control sequence includes note-on events, and wherein each said note-on event includes an intensity value.

4. The apparatus according to claim 1 wherein said input control sequence includes note-off events.

5. The apparatus according to claim 1 further including sound segment directory means for storing sound segment descriptors, wherein each said sound segment descriptor is associated with a selected sound segment in said sound segment storage means, and wherein each said sound segment descriptor includes pointers indicating the location of said selected sound segment in said sound segment storage means.

6. The apparatus according to claim 5 wherein said sound segment storage means stores complete musical phrases and wherein said pointers in said sound segment descriptors indicate locations of sound segments within said complete musical phrases.

7. The apparatus according to claim 5 wherein each said sound segment descriptor includes at least one pitch value describing the pitch of said selected sound segment.

8. The apparatus according to claim 1 wherein said input control sequence includes values describing physical movements of a performer as detected by a musical instrument controller.

9. The apparatus according to claim 1 wherein said plurality of transitions between musical tones includes sound segments corresponding to slurred transitions between musical tones.

10. The apparatus according to claim 9 wherein said sound segment sequencer means selects one of said slurred transitions in response to a pattern of overlapping note-on events from said input control sequence, wherein said overlapping note-on events comprise a first note-on event followed, eventually, by a second note-on event, prior to receiving a note-off event corresponding to said first note-on event.

11. The apparatus according to claim 1 wherein each said sound segment in said sound segment storage means is associated with a musical gesture type, and wherein said sound segment sequencer means further includes:

means for generating a sequence of musical gesture types in response to said input control sequence; and

for each sequential musical gesture type in said sequence of musical gesture types, means for selecting a sound segment from said sound segment storage means, wherein the musical gesture type associated with said sound segment matches said sequential musical gesture type.

12. The apparatus according to claim 11 wherein each said sound segment in said sound segment storage means is further associated with a musical gesture subtype, and wherein said sound segment sequencer means further includes:

means for converting said sequence of musical gesture types into a sequence of musical gesture subtypes by evaluating, for each musical gesture type in said sequence of musical gesture types, a set of conditions based on said input control sequence; and

for each sequential musical gesture subtype in said sequence of musical gesture subtypes, means for selecting a sound segment from said sound segment storage means, wherein the musical gesture subtype associated with said sound segment matches said sequential musical gesture subtype.

13. The apparatus according to claim 11 wherein said gesture types include an attack gesture type, a release gesture type, a transition gesture type, and a sustain gesture type.

14. The apparatus according to claim 12 wherein said gesture subtypes include a hard attack gesture subtype and a soft attack gesture subtype.

15. The apparatus according to claim 12 wherein said gesture subtypes include a large interval slur gesture subtype, and a small interval slur gesture subtype.

16. The apparatus according to claim 12 wherein said gesture subtypes include a slur gesture subtype.

17. The apparatus according to claim 1 wherein said transitions between musical tones include the ending part of a first musical tone and the beginning part of a following musical tone, and wherein any period of silence between the two musical tones is less than 250 milliseconds.

18. The apparatus according to claim 17 wherein said ending part of a first musical tone is associated with a first pitch value and said beginning part of a following musical tone is associated with a second pitch value.

19. The apparatus according to claim 17 wherein said ending part of a first musical tone is associated with a first intensity value and send beginning part of a following musical tone is associated with a second intensity value.

20. The apparatus according to claim 1 wherein said transitions between musical tones include slurred transitions between musical tones.

21. The apparatus according to claim 1 wherein said collection of sound segments further includes a plurality of sustain segments, wherein said sustain segments correspond to a part of a musical tone following an attack or transition segment and preceding a release or subsequent transition section.

22. The apparatus according to claim 21 wherein a selected number of said sustain segments correspond to single vibrato cycles.

23. The apparatus according to claim 1 wherein said sound segment player means further includes means for generating segment events to signal the end of sound segments and to signal a mid-point in said transition segments corresponding to when the pitch begins to change from a beginning pitch to an ending pitch during the transition, and wherein said sound segment sequencer means is further responsive to said to segment events.

24. The apparatus according to claim 1 wherein a first sound segment in said sequence of sound segments is partially played out, up to a stop play location, before switching to a second sound segment in said sequence of sound segments.

25. The apparatus according to claim 24 wherein said second sound segment in said sequence of sound segments is played out beginning at a start play location, and wherein said start play location is offset from the beginning of said second sound segment.

26. The apparatus according to claim 25 wherein said start play location is responsive to said stop play location.

27. The apparatus according to claim 25 wherein a first sound segment is played out partially up to a stop play location, and wherein the following sound segment is played out beginning at a start play location, and wherein said start play location is responsive to a cross-corellation function between the amplitude envelopes of said first sound segment and said following sound segment.

28. The apparatus according to claim 7 wherein said selecting a sequence of sound segments includes, for each selected sound segment, calculating the result of a distance measure between values from said input control sequence on the one hand and values from said sound segment descriptors in said sound segment directory on the other hand, and wherein finding the sound segment descriptor with the minimum distance value from among a selected number of said sound segment descriptors from said sound segment directory means, contributes to said process of selecting a sequence of sound segments.

29. The apparatus according to claim 28 wherein said distance function is responsive to the difference between the pitch value associated with a note-on event in said input control stream and a pitch value associated with a sound segment descriptor in said sound segment directory.

30. The apparatus according to claim 28 wherein a sound segment descriptor in said sound segment directory further includes an intensity value, and wherein said distance function is responsive to the difference between the intensity value associated with a note-on event in said input control stream and an intensity value associated with a sound segment descriptor in said sound segment directory.

31. The apparatus according to claim 28 wherein the difference between the beginning pitch and ending pitch in a transition sound segment corresponds to a sound segment interval value, and wherein the difference between the pitch values associated with two consecutive note-on events in said input control sequence corresponds to an input interval value, and wherein said distance function is responsive to the difference between an input interval value and a sound segment interval value.

32. The apparatus according to claim 1 wherein said sound segment player means further includes means for quickly terminating the playing out of a sound segment, and wherein said means for quickly terminating includes means for smoothly ramping down the amplitude of said sound segment, whereby an audible audio click is avoided.

33. The apparatus according to claim 1 wherein said sound segment player means further includes means for overlapping two sound segments, and wherein said means for overlapping includes means for ramping down the amplitude of a first sound segment while ramping up the amplitude of a following sound segment, whereby a smooth audio cross-fade is implemented between successive sound segments in said sequence of sound segments.

34. The apparatus according to claim 1 wherein said transition sound segments include run transitions, and wherein said run transitions correspond to the transition on between musical tones in a rapid ascending run up sequence of musical tones or a rapid descending run down sequence of musical tones.

35. The apparatus according to claim 1 wherein said sound segments include falloff release sound segments, wherein said falloff release sound segments correspond to downward glissando gestures at the release of a musical tone.

36. The apparatus according to claim 1 wherein said sound segment sequencer means further includes gesture table means for describing musical gesture types and musical gesture subtypes.

37. The apparatus according to claim 36 wherein said sound segment sequencer means further includes a plurality of gesture table means corresponding to different instrumental techniques and playing sytles.

38. The apparatus according to claim 1 wherein said sound segment sequencer means further includes state machine means for executing state transitions in response to said input control sequence, and wherein said state transitions are described by a state transition diagram.

39. The apparatus according to claim 38 wherein said sound segment sequencer means further includes a plurality of state transition diagrams corresponding to different instrumental techniques and playing sytles.

40. The apparatus according to claim 1 wherein said sound segment player means further includes means for pitch-shifting said sound segments.

41. The apparatus according to claim 40 wherein said means for pitch-shifting said sound segments further includes means for pitch-shifting the first part of a transition sound segment differently than the second part of a transition sound segment.

42. The apparatus according to claim 40 wherein said sound segment player means further includes means for intensity-shifting said sound segments.

43. The apparatus according to claim 1 wherein said sound segment player means further includes means for modifying the time duration of said sound segments.

44. The apparatus according to claim 1 wherein said sound segments in said sound segment storage means are encoded as time-domain waveforms.

45. The apparatus according to claim 1 wherein said sound segments in said sound segment storage means are encoded as a sequence spectral coding vectors.

46. The apparatus according to claim 45 wherein said spectral coding vectors include a number of sinusoidal amplitues in combination with indices into a vector quantization codebook.

47. The apparatus according to claim 46 wherein said vector quantization codebook includes time-domain waveforms.

48. The apparatus according to claim 40 wherein said pitch shifting means includes means for estimating the time-varying spectrum of a sound segment based on its time-varying pitch and time-varying intensity.

49. The apparatus according to claim 1 wherein said sound segments in said sound segment storage means are encoded as micro-sequences, and wherein each said micro-sequence includes a list of distinct musical sounds, and wherein each said distinct musical sound has a homogeneous spectral characteristic, or a monotonically changing characteristic.

50. The apparatus according to claim 49 wherein said sound segment player means includes means for individually modifying the duration of each said distinct musical sound in said micro-sequence.

51. The apparatus according to claim 49 wherein said sound segment player means includes means for individually modifying the pitch of each said distinct musical sound in said micro-sequence.

52. A method for synthesizing an ouptut audio signal in response to an input control sequence, comprising:

storing a collection of sound segments in a sound segment storage means, wherein said collection includes a plurality of transitions between musical tones;

generating a sequence of sound segments, selected from said collection of sound segments, in response to said input control sequence, wherein selected ones of said sound segments in said sequence of sound segments correspond to transitions between musical tones; and

playing out and combining said sequence of sound segments to form said output audio signal.

53. The method according to claim 52 wherein each said sound segment in said sound segment storage means is associated with a musical gesture type, and wherein said step of generating a sequence of sound segments further includes the steps of:

generating a sequence of musical gesture types in response to said input control sequence; and

for each sequential musical gesture type in said sequence of musical gesture types, the step of selecting a sound segment from said sound segment storage means, wherein the musical gesture type associated with said sound segment matches said sequential musical gesture type.