Representing speech using MIDI

• US 5,915,237 A
• Filed: 12/13/1996
• Issued: 06/22/1999
• Est. Priority Date: 12/13/1996
• Status: Expired due to Term

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Boss, Dale, Iyengar, Sridhar, Dennis, T. Don
• Primary Examiner(s): Hudspeth, David R.
• Assistant Examiner(s): ABEBE, DANIEL DEMELASH
• Application Number:
  US08/764,933
• Time in Patent Office: 921 Days
• Field of Search: 704/258, 704/260, 704/4, 704/5, 704/272, 704/501
• US Class Current: 704/270.1
• CPC Class Codes: G10H 1/0066 using a MIDI interface
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2240/271 Serial transmission according to any one of RS-232 standards for serial binary single-ended data and control signals between a DTE and a DCE
• CPC Class Codes: G10H 2250/455 Gensound singing voices, i.e. generation of human voices for musical applications, vocal singing sounds or intelligible words at a desired pitch or with desired vocal effects, e.g. by phoneme synthesis
• CPC Class Codes: G10L 19/02 using spectral analysis, e.g. transform vocoders or subband vocoders

Abstract

A speech encoding system for encoding a digitized speech signal into a standard digital format, such as MIDI. The MIDI speech encoding system includes a memory storing a dictionary comprising a digitized pattern and a corresponding segment ID for each of a plurality of speech segments (i.e., phonemes). A speech analyzer identifies each of the segments in the digitized speech signal based on the dictionary. One or more prosodic parameter detectors measure values of the prosodic parameters of each received digitized speech segment. A MIDI speech encoder converts the segment IDs and the corresponding measured prosodic parameter values into a MIDI speech signal. A MIDI speech decoding system includes a MIDI data decoder and a speech synthesizer for converting the MIDI speech signal to a digitized speech signal.

Claims

1. A method of encoding a speech signal into a MIDI compatible format, comprising the steps of:

receiving an analog speech signal, said analog speech signal comprising a plurality of speech segments;

digitizing the analog speech signal;

identifying each of the plurality of speech segments in the received speech signal;

measuring one or more prosodic parameters for each of said identified speech segments; and

converting the speech segment identity and corresponding measured prosodic parameters for each of the identified speech segments into a speech signal having a MIDI compatible format.

2. The method of claim 1 wherein:

said step of receiving comprises the step of receiving an analog speech signal, said analog speech signal comprising a plurality of phonemes;

said step of identifying comprises the step of identifying each of the plurality of phonemes in the received speech signal;

said step of measuring comprises the step of measuring one or more prosodic parameters of each of said identified phonemes; and

said step of converting comprises the step of converting the phoneme identity and corresponding measured prosodic parameters for each identified phoneme into a MIDI speech signal, said MIDI speech signal comprising a plurality of MIDI messages that represents the analog speech signal.

3. The method of claim 2 and further comprising the step of storing the MIDI speech signal to enable the later playback of said analog speech signal using said stored MIDI speech signal.

4. The method of claim 2 and further comprising the step of communicating the MIDI speech signal over a transmission medium.

5. The method of claim 4 wherein said step of communicating the MIDI speech signal comprises the step of communicating the MIDI speech signal to a remote user via the Internet.

6. The method of claim 4 wherein said step of communicating further comprises the step of communicating a voice font ID identifying a designated output voice font to be used during playback or reconstruction of the analog speech signal using said MIDI speech signal.

7. The method of claim 2 and further comprising the step of:

storing a dictionary comprising a digitized phoneme pattern and an associated phoneme ID for each said phoneme;

said step of identifying comprising the steps of comparing the digitized speech signal to the phoneme patterns stored in the dictionary to identify the phonemes in the digitized speech signal.

8. The method of claim 2 and further comprising the step of:

storing a dictionary comprising a digitized phoneme pattern and an associated MIDI compatible phoneme identifier for each said phoneme;

said step of identifying comprising the steps of comparing the digitized speech signal to the patterns stored in the dictionary to identify the phonemes in the digitized speech signal.

9. The method of claim 8, wherein said step of storing a dictionary comprises storing a dictionary comprising, for each of said phonemes, a digitized phoneme pattern and a MIDI channel number associated with each said phoneme.

10. The method of claim 8, wherein said step of storing a dictionary comprises storing a dictionary comprising, for each of said phonemes, a digitized phoneme pattern and a MIDI program number associated with each said phoneme.

11. The method of claim 7 wherein said step of measuring one or more prosodic parameters for each of said phonemes comprises the steps of:

measuring the pitch for each of said phonemes;

measuring the duration for each of said phonemes; and

measuring the amplitude for each of said phonemes.

12. The method of claim 11, wherein said step of converting comprises the steps of:

converting the phoneme ID of each identified phoneme into a MIDI compatible identifier that identifies the phoneme;

converting the measured pitch of each identified phoneme into a MIDI note number;

converting the measured amplitude of each identified phoneme into a MIDI velocity number;

generating one or more MIDI Note On and Note Off messages for each identified phoneme based on the measured duration of the segment.

13. The method of claim 11, wherein said step of converting comprises the steps of:

converting the phoneme ID of each identified phoneme into a MIDI compatible identifier that identifies the phoneme;

converting the measured pitch of each identified phoneme into a MIDI note number;

converting the measured amplitude of each identified phoneme into a MIDI velocity number;

generating, for each said identified phoneme, a MIDI Note On command at a MIDI velocity specified by the corresponding MIDI velocity number to turn on the phoneme, and a MIDI Note On command at a velocity of zero to turn off the phoneme based on the measured duration of the segment.

14. The method of claim 13 wherein said step of converting the phoneme ID comprises the step converting the phoneme ID of each identified segment into a corresponding MIDI channel number.

15. The method of claim 10 wherein said step of measuring one or more prosodic parameters for each of said phonemes comprises the steps of:

measuring the pitch for each of said phonemes;

measuring the duration for each of said phonemes; and

measuring the amplitude for each of said phonemes.

16. The method of claim 15, wherein said step of converting comprises the steps of:

identifying the MIDI program associated with each said identified phoneme using said dictionary;

converting the measured pitch of each identified phoneme into a MIDI note number;

converting the measured amplitude of each identified phoneme into a MIDI velocity number;

generating one or more MIDI Note On and Note Off commands for each identified phoneme based on the measured duration of the phoneme.

17. The method of claim 16, further comprising the step of outputting the MIDI speech signal, said MIDI speech signal comprising information identifying, for each of the identified phonemes, the MIDI program associated with the phoneme, the MIDI note number for each identified phoneme, and the MIDI velocity number for each identified phoneme, and one or more MIDI Note On and Note Off messages.

18. The method of claim 1 and further comprising the steps of:

storing a designated input voice font, said input voice font comprising a plurality of digitized segments, each voice font segment having a plurality of corresponding prosodic parameters;

said step of measuring one or more prosodic parameters comprising the steps of;

measuring the prosodic parameters of the received digitized speech segments; and

comparing values of the measured prosodic parameters of the received digitized speech segments to values of the prosodic parameters of the segments of the designated input voice font.

19. A method of generating an analog speech signal based on a speech signal in a MIDI compatible format, said method comprising the steps of:

storing a dictionary comprising;

a) a digitized pattern for each of a plurality of speech segments; and

b) a corresponding segment ID identifying each of the digitized segment patterns;

receiving a speech signal in a MIDI compatible format;

decoding the received speech signal in the MIDI compatible format;

converting the received speech signal in the MIDI compatible format into a plurality of speech segment IDs and corresponding prosodic parameter values;

selecting speech segment patterns in the dictionary corresponding to the speech segment IDs in the converted received speech signal;

modifying the selected speech segment patterns according to the values of the corresponding prosodic parameters in the converted received speech signal;

outputting the modified segment patterns to generate a digitized speech signal; and

converting the outputted digitized speech signal to an analog format.

20. The method of claim 19 wherein, said dictionary comprises:

a) a digitized pattern for each of a plurality of speech segments; and

b) a corresponding MIDI program number for each of the speech segment patterns.

21. The method of claim 20 wherein said step of receiving comprises the step of receiving a MIDI speech signal, said MIDI speech signal comprising a plurality of MIDI program numbers identifying a MIDI program for each of a plurality of speech segments, MIDI note numbers, MIDI velocity numbers, and one or more MIDI Note ON and Note Off messages.

22. The method of claim 21 wherein said step of decoding comprises the step of identifying the MIDI program numbers, MIDI note numbers, MIDI velocity numbers, and one or more status bytes in the received MIDI speech signal.

23. The method of claim 22 wherein said step of converting the MIDI speech signal comprises the steps of:

identifying, using said dictionary, the speech segment patterns corresponding to the MIDI program numbers in the received MIDI compatible speech signal;

converting each MIDI note number in the received MIDI speech signal to a corresponding pitch value;

converting each MIDI velocity number in the received MIDI speech signal to a corresponding amplitude value; and

determining a duration value for each identified speech segment pattern based on the one or more MIDI Note On and Note Off messages and one or more MIDI timing messages in the received MIDI speech signal.

24. The method of claim 23 wherein said step of selecting speech segment patterns in the dictionary comprises the step of selecting, using said dictionary, the speech segment patterns corresponding to the MIDI program numbers in the received MIDI speech signal.

25. The method of claim 24 wherein said step of modifying comprises the step of:

modifying the pitch, amplitude and duration of each selected speech segment pattern according to the corresponding pitch value, amplitude value and duration value, respectively.

26. A computer-readable medium having stored thereon a plurality of instructions including instructions, when executed by a processor result in:

identifying and analyzing each of a plurality of speech segments in a digitized speech signal;

measuring a plurality of prosodic parameters for each said identified speech segment, said prosodic parameters comprising at least pitch and amplitude;

converting the measured prosodic parameters to corresponding MIDI compatible values relating to prosody, including converting each measured pitch value to a corresponding MIDI note number and converting each measured amplitude value to a corresponding MIDI velocity number; and

generating a MIDI speech signal comprising an identification of each identified speech segment and the corresponding MIDI compatible values relating to prosody.

27. A computer-readable medium having stored thereon a plurality of instructions including instructions, when executed by a processor result in:

analyzing a MIDI compatible speech signal, said MIDI compatible speech signal comprising a plurality of speech segment IDs and corresponding MIDI compatible values relating to prosody;

identifying the plurality of speech segment IDs and corresponding MIDI compatible values relating to prosody in the MIDI speech signal;

selecting a digitized speech segment pattern stored in memory corresponding to each of the identified speech segment IDs;

modifying the selected digitized speech segment patterns according to the MIDI compatible values relating to prosody;

outputting the modified speech segment patterns to generate a digitized speech signal.

28. An apparatus for encoding an analog speech signal into a MIDI speech signal comprising:

a memory storing a dictionary comprising a digitized pattern and a corresponding segment ID for each of a plurality of speech segments;

an A/D converter having an input adapted for receiving an analog speech signal and providing a digitized speech signal output;

a speech analyzer coupled to said memory and said A/D converter, said speech analyzer adapted to receive a digitized speech signal and identify each of the segments in the digitized speech signal based on said dictionary, said speech analyzer adapted to output the segment ID for each of said identified speech segments;

one or more prosodic parameter detectors coupled to said memory and said speech analyzer, said detectors adapted to measure values of the prosodic parameters of each received digitized speech segment; and

a MIDI speech encoder coupled to said speech analyzer and said prosodic parameter detectors, said MIDI speech encoder adapted to convert a segment ID and the measured values of the corresponding measured prosodic parameters for each of a plurality of speech segments into a MIDI speech signal.

29. An apparatus for generating a speech signal from a MIDI speech signal, said apparatus comprising:

a MIDI data decoder adapted to receive and decode a MIDI speech signal comprising MIDI compatible speech segment IDs and corresponding MIDI compatible values relating to prosody;

a memory adapted to a store a dictionary, said dictionary comprising a plurality of speech segment patterns and speech segment IDs for a plurality of speech segments;

a speech synthesizer coupled to the MIDI data decoder and the memory, said speech synthesizer selecting a digitized speech segment pattern stored in the dictionary corresponding to each of the speech segment IDs on the received MIDI compatible speech signal, modifying the selected digitized speech segment patterns according to the MIDI compatible values relating to prosody, and outputting the modified speech segment patterns to generate a digitized speech signal.

30. A computer for encoding a speech signal into a MIDI signal comprising:

a CPU;

an audio input device adapted to receive an analog speech signal and having an output;

an A/D converter having an input coupled to the output of said audio input device and providing a digitized speech signal output, said converter output coupled to said CPU;

a memory coupled to said CPU, said memory storing a dictionary comprising a digitized speech segment pattern and a corresponding segment ID for each of a plurality of speech segments; and

said CPU being adapted to;

identify, using said dictionary, each of a plurality of speech segments in a received digitized speech signal;

measure one or more prosodic parameters for each of the identified segments; and

encode the speech segment ID of each identified speech segment and the corresponding measured prosodic parameters into a MIDI signal.