System and method for voice-to-voice conversion

• US 10,614,826 B2
• Filed: 05/24/2018
• Issued: 04/07/2020
• Est. Priority Date: 05/24/2017
• Status: Active Grant

• Current Assignee: Modulate, Inc.
• Original Assignee: Modulate, Inc.
• Inventors: Huffman, William Carter, Pappas, Michael
• Primary Examiner(s): Singh, Satwant K
• Application Number:
  US15/989,062
• Publication Number:
  US 20180342256A1
• Time in Patent Office: 684 Days
• Field of Search: None
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2021/0135 Voice conversion or morphing
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: G10L 25/30 using neural networks

Abstract

A method of building a speech conversion system uses target information from a target voice and source speech data. The method receives the source speech data and the target timbre data, which is within a timbre space. A generator produces first candidate data as a function of the source speech data and the target timbre data. A discriminator compares the first candidate data to the target timbre data with reference to timbre data of a plurality of different voices. The discriminator determines inconsistencies between the first candidate data and the target timbre data. The discriminator produces an inconsistency message containing information relating to the inconsistencies. The inconsistency message is fed back to the generator, and the generator produces a second candidate data. The target timbre data in the timbre space is refined using information produced by the generator and/or discriminator as a result of the feeding back.

Claims

1. A method of building a speech conversion system using target voice information from a target voice, and speech data that represents a speech segment of a source voice, the method comprising:

receiving source speech data that represents a first speech segment of a source voice;

receiving target timbre data relating to the target voice, the target timbre data being within a timbre space;

using a generative machine learning system to produce first candidate speech data that represents a first candidate speech segment in a first candidate voice as a function of the source speech data and the target timbre data;

using a discriminative machine learning system to compare the first candidate speech data to the target timbre data with reference to timbre data of a plurality of different voices,

said using the discriminative machine learning system comprising determining at least one inconsistency between the first candidate speech data and the target timbre data with reference to the timbre data of the plurality of different voices, the discriminative machine learning system producing an inconsistency message having information relating to the inconsistency between the first candidate speech data and the target timbre data;

feeding back the inconsistency message to the generative machine learning system;

using the generative machine learning system to produce second candidate speech data, that represents a second candidate speech segment in a second candidate voice, as a function of the inconsistency message; and

refining the target timbre data in the timbre space using information produced by the generative machine learning system and/or discriminative machine learning system as a result of said feeding back.

2. The method as defined by claim 1, wherein the source speech data is from an audio input of the source voice.

3. The method as defined by claim 1, wherein the second candidate speech segment provides a higher probability that of being from the target voice than the first candidate speech segment.

4. The method as defined by claim 1, further comprising transforming the source speech data to into the target timbre.

5. The method as defined by claim 1, wherein the target timbre data is obtained from an audio input in the target voice.

6. The method as defined by claim 1, wherein the machine learning system is a neural network.

7. The method as defined by claim 1, further comprising:

mapping a representation of the plurality of voices and the first candidate voice in a vector space as a function of a frequency distribution in the speech segment provided by each voice.

8. The method as defined by claim 7, further comprising:

adjusting a representation of the first candidate voice relative to representations of the plurality of voices in the vector space to reflect the second candidate voice as a function of the inconsistency message.

9. The method as defined by claim 1, wherein the inconsistency message is produced when the discriminative neural network has less than a 95 percent confidence interval that the first candidate voice is the target voice.

10. The method as defined by claim 1, further comprising:

assigning an identity to the candidate voice by comparing the candidate voice to the plurality of voices.

11. The method as defined by claim 1, wherein the plurality of voices are in a vector space.

12. The method as defined by claim 1, wherein the target timbre data is filtered by a temporal receptive field.

13. The method as defined by claim 1, further comprising using the generative machine learning system to produce a final candidate speech segment in a final candidate voice, as a function of a null inconsistency message,

the final candidate speech segment mimicking the first speech segment in the target timbre.

14. The method as defined by claim 13, wherein the temporal receptive field is between about 10 milliseconds and about 1000 milliseconds.

15. The method as defined by claim 1, further comprising means for extracting the target timbre data from a target speech segment.

16. A system for training a speech conversion system, the system comprising:

source speech data that represents a first speech segment of a source voice;

target timbre data that relates to a target voice;

a generative machine learning system configured to produce first candidate speech data that represents a first candidate speech segment in a first candidate voice as a function of the source speech data and the target timbre data;

a discriminative machine learning system configured to;

compare the first candidate speech data to the target timbre data with reference to timbre data of a plurality of different voices, and

determine whether there is at least one inconsistency between the first candidate speech data and the target timbre data with reference to the timbre data of the plurality of different voices, and when the at least one inconsistency exists;

produce an inconsistency message having information relating to the inconsistency between the first candidate speech data and the target timbre data, and

provide the inconsistency message back to the generative machine learning system.

17. The system as defined by claim 16, wherein the generative machine learning system is configured to produce a second candidate speech segment as a function of the inconsistency message.

18. The system as defined by claim 16, wherein the machine learning system is a neural network.

19. The system as defined by claim 16, further comprising:

a vector space configured to map a representation of the plurality of voices, including the candidate voice, as a function of a frequency distribution in the speech segment provided by each voice.

20. The system as defined by claim 19, wherein a voice feature extractor is configured to adjust a representation of the candidate voice relative to representations of the plurality of voices in the vector space to update and reflect the second candidate voice as a function of the inconsistency message.

21. The system as defined by claim 16, wherein the candidate voice is distinguished from the target voice when the discriminative neural network has less than a 95 percent confidence interval.

22. The system as defined by claim 16, wherein the discriminative machine learning system is configured to determine the identity of the speaker of the candidate voice by comparing the first or second candidate voice to the plurality of voices.

23. The system as defined by claim 16, further comprising a vector space configured to contain a plurality of voices.

24. The system as defined by claim 16, wherein the generative machine learning system is configured to produce a final candidate speech segment in a final candidate voice, as a function of a null inconsistency message,

the final candidate speech segment mimicking the first speech segment as the target voice.

25. The system as defined by claim 16, wherein the target timbre data is filtered by a temporal receptive field.

26. The system as defined by claim 25, wherein the temporal receptive field is between about 10 milliseconds and about 2,000 milliseconds.

27. The system as defined by claim 16, wherein the source speech data is from a source audio input.

28. A computer program product for use on a computer system for training a speech conversion system using source speech data that represents a speech segment from a source voice for conversion into an output voice having a target voice timbre, the computer program product comprising a tangible, non-transient computer usable medium having computer readable program code thereon, the computer readable program code comprising:

program code for causing a generative machine learning system to produce first candidate speech data that represents a first candidate speech segment in a first candidate voice as a function of the source speech data and target timbre data;

program code for causing a discriminative machine learning system to compare the first candidate speech data to the target timbre data with reference to the timbre data of the plurality of different voices;

program code for causing the discriminative machine learning system to determine at least one inconsistency between the first candidate speech data and the target timbre data with reference to the timbre data of the plurality of different voices;

program code for causing the discriminative machine learning system to produce an inconsistency message having information relating to the inconsistency between the first candidate speech data and the target timbre data with reference to the timbre data of the plurality of different voices;

program code for causing the discriminative machine learning system to feed the inconsistency message back to the generative machine learning system; and

program code for causing the generative machine learning system to produce second candidate speech data representing a second candidate speech segment in a second candidate voice as a function of the inconsistency message.

29. The computer program product as defined by claim 28, further comprising:

program code for extracting the target timbre data from a target audio input.

30. The computer program product as defined by claim 28, wherein the machine learning system is a neural network.

31. The computer program product as defined by claim 28, further comprising:

program code for mapping a representation of each of the plurality of voices and the candidate voice in a vector space as a function of the timbre data from each voice.

32. The computer program product as defined by claim 31, further comprising:

program code for adjusting the representation of the candidate voice relative to at least one representation of the plurality of voices in the vector space, to update and reflect the second candidate voice as a function of the inconsistency message.

33. The computer program product as defined by claim 28, further comprising:

program code for assigning a speaker identity to the candidate voice by comparing the candidate voice to the plurality of voices.

34. The computer program product as defined by claim 28, further comprising:

program code for filtering an inputted target audio using a temporal receptive field to produce the timbre data.

35. The computer program product as defined by claim 34, wherein the temporal receptive field is between about 10 milliseconds and about 2,000 milliseconds.

36. The computer program product as defined by claim 28, further comprising:

program code for converting the source speech data that represents the speech segment from the source voice into a transformed speech segment in the target timbre.

37. The computer program product as defined by claim 36, further comprising:

program code for adding a watermark to the transformed speech segment.