System and method for creating timbres

• US 10,622,002 B2
• Filed: 05/24/2018
• Issued: 04/14/2020
• Est. Priority Date: 05/24/2017
• Status: Active Grant

• Current Assignee: Modulate, Inc.
• Original Assignee: Modulate, Inc.
• Inventors: Huffman, William Carter, Pappas, Michael
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US15/989,072
• Publication Number:
  US 20180342258A1
• Time in Patent Office: 691 Days
• Field of Search: 84603, 84634, 84622, 381 17, 434307 A, 434169, 434236, 455307, 455418, 600544, 600559, 704 9, 704207, 704232, 704235, 704258, 704267, 704268, 704500, 704 3, 704214, 704219, 704220, 704231, 704246, 704254, 704260, 704206, 704269, 704270, 704272, 715752, 345619, 707769, 709219
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2021/0135 Voice conversion or morphing
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: G10L 25/30 using neural networks

Abstract

A method of building a new voice having a new timbre using a timbre vector space includes receiving timbre data filtered using a temporal receptive field. The timbre data is mapped in the timbre vector space. The timbre data is related to a plurality of different voices. Each of the plurality of different voices has respective timbre data in the timbre vector space. The method builds the new timbre using the timbre data of the plurality of different voices using a machine learning system.

Claims

1. A method of building a new voice having a new timbre using a timbre vector space, the method comprising:

receiving timbre data filtered using a temporal receptive field, the timbre data being mapped in the timbre vector space, the timbre data being related to a plurality of different voices, each of the plurality of different voices having respective timbre data in the timbre vector space; and

building, using a machine learning system, the new voice having the new timbre, using the timbre data of the plurality of different voices.

2. The method as defined by claim 1, further comprising means for filtering the timbre data.

3. The method as defined by claim 1, further comprising providing source speech;

and

converting the source speech to the new timbre while maintaining source cadence and source accent.

4. The method as defined by claim 1, further comprising:

receiving a new speech segment from a new voice;

filtering the new speech segment into a new analytical audio segment; and

building, by mapping, the new voice in the vector space relative to a plurality of mapped voices.

5. The method as defined by claim 1, further comprising:

using a generative neural network to produce a first candidate speech segment, in a candidate voice, as a function of a mathematical operation between a first voice and a second voice.

6. The method as defined by claim 1, wherein a cluster of voice representations in the timbre vector space are representative of a particular accent.

7. The method as defined by claim 1, wherein the timbre data is extracted from a speech segment provided by each of the plurality of different voices.

8. The method as defined by claim 1, wherein the respective timbre data includes information that relates to a respective voice.

9. The method as defined by claim 4, wherein the timbre data is mapped in the timbre vector space independent of any specific sound.

10. The method as defined by claim 4, further comprising refining at least one of the characteristics of the new voice on the basis of the relation of the new voice to the plurality of mapped voices.

11. The method as defined by claim 4, wherein the filtering is performed by a neural network.

12. The method as defined by claim 4, wherein the mapping is performed by a neural network.

13. The method as defined by claim 1, wherein the plurality of different voices are human voices.

14. A system that produces a new target voice using a timbre vector space, the system comprising:

a timbre vector space configured to store timbre data;

timbre data filtered using the temporal receptive field, the timbre data being related to a plurality of different voices; and

a machine learning system configured to create the new target voice using the timbre data.

15. The system as defined by claim 14, wherein the machine learning system is a neural network.

16. The system as defined by claim 14, wherein the machine learning system is configured to:

receive a new speech segment from a new voice,

filter the new speech segment into new timbre data,

map the new timbre data in the vector space relative to a plurality of timbre data, and

determine at least one voice characteristic of the new voice on the basis of the relation of the new timbre data relative to the plurality of timbre data.

17. The system as defined by claim 14, wherein converting the timbre data to the new target voice is initiated by performing a mathematical operation using at least one voice characteristic of the timbre data as a variable.

18. The system as defined by claim 14, wherein a cluster of voice representations in the timbre vector space are representative of a particular accent.

19. A computer program product for use on a computer system for producing a new target voice using a timbre vector space, the computer program product comprising a tangible, non-transient computer usable medium having computer readable program code thereon, the computer readable program code comprising:

program code for receiving timbre data filtered using a temporal receptive field, the timbre data being stored in the timbre vector space, the timbre data being related to a plurality of different voices; and

program code for creating, using a machine learning system, the new target voice using the timbre data.

20. The program code as defined by claim 19, further comprising:

program code for receiving a new speech segment from a new voice;

program code for causing the machine learning system to filter the new speech segment into a new analytical audio segment;

program code for mapping the new voice in the vector space relative to a plurality of mapped voices; and

program code for determining at least one of the characteristics of the new voice on the basis of the relation of the new voice to the plurality of mapped voices.

21. The program code as defined by claim 19, wherein the machine learning system is a neural network.

22. The program code as defined by claim 19, wherein converting the timbre data to the new target voice is initiated by performing a mathematical operation using at least one voice characteristic of the timbre data as a variable.

23. The program code as defined by claim 19, wherein a cluster of voice representations in the timbre vector space are representative of a particular accent.

24. A method of converting a speech segment from a source timbre to a target timbre, the method comprising:

storing timbre data related to a plurality of different voices, each of the plurality of different voices having respective timbre data in a timbre vector space, the timbre data having been filtered using a temporal receptive field and mapped in the timbre vector space;

receiving a source speech segment in a source voice to transform into a target voice;

receiving a selection of a target voice, the target voice having a target timbre, the target voice being mapped in the timbre vector space with reference to the plurality of different voices;

transforming, using a machine learning system, the source speech segment from the timbre of the source voice to the timbre of the target voice.