Generating dynamically controllable composite data structures from a plurality of data segments

• US 9,147,166 B1
• Filed: 08/10/2012
• Issued: 09/29/2015
• Est. Priority Date: 08/10/2011
• Status: Active Grant

• Current Assignee: Konlanbi
• Original Assignee: Konlanbi
• Inventors: Drame, Cyril
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Hanchak, Walter
• Application Number:
  US13/572,352
• Time in Patent Office: 1,145 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A method of producing dynamic controllable data composites from two or more data segments includes: building or training one or more function mappers to map between one or more extracted feature envelopes sets from the original data and one or more general parametric representations of the data; combining the extracted feature envelopes or the function mappers using two or more audio segments; and feeding the extracted feature envelopes or combined feature envelopes to the function mappers to obtain synthesis parameters to drive a synthesis process.

Claims

1. A method of producing dynamically controllable data composites from two or more data segments, the method comprising:

training one or more function mappers to map between one or more feature envelopes sets that are extracted from data segments of audio data and one or more corresponding parametric representations of the audio data by using at least one training parameter to discriminate between the data segments in a training process for the one or more function mappers, the one or more feature envelope sets corresponding to one or more perceptual characteristics of the audio data, and the one or more parametric representations corresponding to one or more spectral characteristics of the audio data;

forming a composite data structure by using at least one control parameter to combine values for a plurality of the data segments, each of the plurality of data segments being associated with a corresponding feature envelope set and a corresponding function mapper; and

generating synthesis parameters for the composite data structure by feeding the feature envelopes sets corresponding to the composite data structure to the corresponding one or more function mappers and combining outputs of the one or more function mappers in accordance with the composite data structure, the synthesis parameters being used to drive a synthesis process.

2. The method of claim 1, wherein training the one or more function mappers includes combining two or more feature envelope sets at an input level of a first function mapper.

3. The method of claim 1, wherein training the one or more function mappers includes combining two or more feature envelope sets at a body level of a first function mapper by training the first function mapper with a first training parameter that operates as a discriminant input on the two or more feature envelope sets.

4. The method of claim 1, wherein generating the synthesis parameters includes combining values at output levels of a first function mapper and a second function mapper, each of the first and second function mappers being fed with two or more feature envelope sets.

7. The method of claim 1, wherein the one or more perceptual characteristics of the audio data include at least one of pitch, loudness or brightness.

8. The method of claim 1, wherein the one or more spectral characteristics of the audio data include values for spectral peaks in one or more of the data segments.

9. The method of claim 1, wherein generating the synthesis parameters includes using a first control parameter to interpolate values between output levels of a first function mapper and a second function mapper or to extrapolate values from the output levels of the first function mapper and the second function mapper.

10. The method of claim 1, wherein generating the synthesis parameters includes using a first control parameter to calculate a linear combination of values from the output levels of a first function mapper and a second function mapper.

11. The method of claim 1, wherein the one or more function mappers include at least one neural network that is trained adaptively in the training process for the one or more function mappers.

12. The method of claim 1, wherein the at least one training parameter includes a discriminant input that is used to combine two or more of the data segments in a first function mapper.

13. The method of claim 1, further comprising:

determining boundaries of the data segments in the audio data from changes in perceptual characteristics in the audio data;

associating one or more categories with the data segments, each category corresponding to perceptual descriptor for a corresponding data segment; and

training the one or more function mappers by using the one or more categories in association with the at least one training parameter to discriminate between the data segments.

14. The method of claim 1, further comprising:

providing characteristics of a first segment of the audio data to a user interface;

receiving from the user interface values for a first perceptual descriptor for the first segment;

associating the first perceptual descriptor with the first data segment; and

training a first function mapper by using the first perceptual descriptor in association with a first training parameter to discriminate between the first data segment and other data segments.

5. An apparatus for producing dynamically controllable data composites from two or more data segments, the apparatus including at least one computer that performs operations for computer-implemented modules including:

a training module for training one or more function mappers to map between one or more feature envelopes sets that are extracted from data segments of audio data and one or more corresponding parametric representations of the audio data by using at least one training parameter to discriminate between the data segments in a training process for the one or more function mappers, the one or more feature envelope sets corresponding to one or more perceptual characteristics of the audio data, and the one or more parametric representations corresponding to one or more spectral characteristics of the audio data;

a data-structure module for forming a composite data structure by using at least one control parameter to combine values for a plurality of the data segments, each of the plurality of data segments being associated with a corresponding feature envelope set and a corresponding function mapper; and

a synthesis module for generating synthesis parameters for the composite data structure by feeding the feature envelopes sets corresponding to the composite data structure to the corresponding one or more function mappers and combining outputs of the one or more function mappers in accordance with the composite data structure, the synthesis parameters being used to drive a synthesis process.

15. The apparatus of claim 5, wherein the at least one training parameter includes a discriminant input that is used to combine two or more of the data segments at an input level of a first function mapper.

16. The apparatus of claim 5, wherein the training module is further configured to perform operations including:

determining boundaries of the data segments in the audio data from changes in perceptual characteristics in the audio data;

associating one or more categories with the data segments, each category corresponding to perceptual descriptor for a corresponding data segment; and

training the one or more function mappers by using the one or more categories in association with the at least one training parameter to discriminate between the data segments.

17. The apparatus of claim 5, wherein the training module is further configured to perform operations including:

providing characteristics of a first segment of the audio data to a user interface;

receiving from the user interface values for a first perceptual descriptor for the first segment;

associating the first perceptual descriptor with the first data segment; and

training a first function mapper by using the first perceptual descriptor in association with a first training parameter to discriminate between the first data segment and other data segments.

6. A non-transitory computer-readable medium that stores a computer program for producing dynamically controllable data composites from two or more data segments, the computer program including instructions that, when executed by a computer, cause the computer to perform operations including:

training one or more function mappers to map between one or more feature envelopes sets that are extracted from data segments of audio data and one or more corresponding parametric representations of the audio data by using at least one training parameter to discriminate between the data segments in a training process for the one or more function mappers, the one or more feature envelope sets corresponding to one or more perceptual characteristics of the audio data, and the one or more parametric representations corresponding to one or more spectral characteristics of the audio data;

forming a composite data structure by using at least one control parameter to combine values for a plurality of the data segments, each of the plurality of data segments being associated with a corresponding feature envelope set and a corresponding function mapper; and

generating synthesis parameters for the composite data structure by feeding the feature envelopes sets corresponding to the composite data structure to the corresponding one or more function mappers and combining outputs of the one or more function mappers in accordance with the composite data structure, the synthesis parameters being used to drive a synthesis process.

18. The non-transitory computer-readable medium of claim 6, wherein training the one or more function mappers includes combining two or more feature envelope sets at an input level of a first function mapper.

19. The non-transitory computer-readable medium of claim 6, wherein training the one or more function mappers includes combining two or more feature envelope sets at a body level of a first function mapper by training the first function mapper with a first training parameter that operates as a discriminant input on the two or more feature envelope sets.

20. The non-transitory computer-readable medium of claim 6, wherein generating the synthesis parameters includes combining values at output levels of a first function mapper and a second function mapper, each of the first and second function mappers being fed with two or more feature envelope sets.