SYSTEMS AND METHODS FOR AUTOMATIC-CREATION OF SOUNDTRACKS FOR SPEECH AUDIO

• US 20180032610A1
• Filed: 07/28/2017
• Published: 02/01/2018
• Est. Priority Date: 07/29/2016
• Status: Active Grant

• Current Assignee: Booktrack Holdings Limited
• Original Assignee: Booktrack Holdings Limited
• Inventors: Cameron, Paul Charles, Wilson, Craig Andrew, De Vocht, Petrus Matheus Godefridus, Moore, Brock David
• Granted Patent: US 10,698,951 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G09B 5/062 Combinations of audio and printed presentations, e.g. magnetically striped cards, talking books, magnetic tapes with printed texts thereon
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals

Abstract

A method of automatically generating a digital soundtrack intended for synchronised playback with associated speech audio, the method executed by a processing device or devices having associated memory. The method comprises syntactically and/or semantically analysing text representing or corresponding to the speech audio at a text segment level to generate an emotional profile for each text segment in the context of a continuous emotion model. The method further comprises generating a soundtrack for the speech audio comprising one or more audio regions that are configured or selected for playback during corresponding speech regions of the speech audio, and wherein the audio configured for playback in the audio regions is based on or a function of the emotional profile of one or more of the text segments within the respective speech regions.

Claims

1. A method of automatically generating a digital soundtrack intended for synchronised playback with associated speech audio, the soundtrack comprising one or more audio regions configured for synchronised playback with corresponding speech regions of the speech audio, the method executed by a processing device or devices having associated memory, the method comprising:

(a) receiving or retrieving raw text data representing or corresponding to the speech audio into memory;

(b) applying natural language processing (NLP) to the raw text data to generate processed text data comprising token data that identifies individual tokens in the raw text, the tokens at least identifying distinct words or word concepts;

(c) applying semantic analysis to a series of text segments of the processed text data based on a continuous emotion model defined by a predefined number of emotional category identifiers each representing an emotional category in the model, the semantic analysis being configured to parse the processed text data to generate, for each text segment, a segment emotional data profile based on the continuous emotion model;

(d) identifying a series of text or speech regions comprising a text segment or a plurality of adjacent text segments having an emotional association by processing the segment emotional data profiles of the text segments with respect to predefined rules, and generating audio region data defining the intended audio regions corresponding to the identified text regions, each audio region being defined by a start position indicative of the position in the text at which the audio region is to commence playback, a stop position indicative of the position in the text at which the audio region is to cease playback, and a generated region emotional profile based on the segment emotional profiles of the text segments within its associated text region;

(e) processing an accessible audio database or databases comprising audio data files and associated audio profile information to select an audio data file for playback in each audio region, the selection at least partly based on the audio profile information of the audio data file corresponding to the region emotional profile of the audio region, and defining audio data for each audio region representing the selected audio data file for playback; and

(f) generating soundtrack data representing the created soundtrack for synchronised playback with the speech audio, the soundtrack data comprising data representing the generated audio regions and audio data associated with those audio regions.

2. The method according to claim 1 wherein the speech audio is pre-recorded or pre-generated.

3. The method according to claim 1 wherein the speech audio is an audiobook and the speech regions correspond to narration regions in the audiobook, and wherein the soundtrack comprises a soundtrack timeline along which the one or more audio regions are configured for synchronised playback with corresponding narration regions in the audiobook playback timeline, each audio region having a position along the soundtrack timeline that is dynamically adjustable to maintain synchronization of the audio regions of the soundtrack with their respective narration regions in the audiobook based on a narration speed variable indicative of the playback narration speed of the audiobook.

4. The method according to claim 1 wherein step (a) of receiving or retrieving raw text data representing the speech audio into memory comprises:

receiving or retrieving a digital audio file or files representing the speech audio; and

processing the digital audio file or files with one or more speech-to-text engines to generate raw text data representing the speech audio.

5. The method according to claim 1 wherein the step (a) of receiving or retrieving raw text data representing the speech audio into memory comprises:

receiving identification data indicative of the speech audio;

receiving or retrieving raw text data representing the speech audio from one or more electronic text or content sources based on the identification data.

6. The method according to claim 1 wherein the continuous emotion model is further defined by lexicon data representing a set of lexicons for the emotional category identifiers, each lexicon comprising data indicative of a list of words and/or word concepts that are categorised or determined as being associated with the emotional category identifier associated with the lexicon.

7. The method according to claim 6 wherein each word or word concept in each lexicon has one or more associated lexicon attribute values or identifiers.

8. The method according to claim 7 wherein each word or word concept has a weighting value representing how strongly the word or word concept represents the emotional category identifier of the lexicon, and a syntactic value or identifier representing the part of speech or type of word or word concept.

9. The method according to claim 6 wherein applying semantic analysis to the processed text data comprises generating segment emotional data profiles that represent for each emotional category identifier the absolute count or frequency of tokens in the text segment corresponding to the associated lexicon.

10. The method according to claim 6 wherein each segment emotional data profile is generated in the form of a multi-dimensional data set comprising a plurality of count or frequency values, one for each emotional category identifier, and wherein the count or frequency values are based on the number of tokens in the text segment corresponding to or matching to their lexicon associated with an emotional category identifier.

11. The method according to claim 10 wherein the multi-dimensional data sets of the segment emotional data profiles are multi-dimensional representations of the emotional profile of the text segments, each emotional category identifier representing a distinct dimension within the continuous emotion model.

12. The method according to claim 10 wherein the generated segment emotional data profiles comprise a multi-dimensional data set comprising absolute count values or frequency values for each emotional category identifier.

13. The method according to claim 12 wherein the absolute count values or frequency values of the multi-dimensional data set represent or are indicative of the absolute intensity of the text segment for each emotional category identifier.

14. The method according to claim 12 wherein the method further comprises generating baseline statistical values for each emotional category identifier across the entire text, and normalising or scaling the segment emotional data profiles based on or as a function of the generated baseline statistical values to generate relative segment emotional data profiles.

15. The method according to claim 1 wherein the continuous emotion model comprises a 2-dimensional circular reference frame defined by a circular perimeter or boundary extending about a central origin, with each emotional category identifier represented by a segment or spoke of the circular reference frame to create a continuum of emotions.

16. The method according to claim 15 wherein the segment emotional data profiles are multi-dimensional data sets comprising absolute or relative intensity values for each emotional category identifier, and the method comprises mapping each data set as multiple two-dimensional vectors into the circular reference frame, a vector representing the intensity value of each emotional category identifier of a text segment.

17. The method according to claim 15 wherein the segment emotional data profiles are multi-dimensional data sets comprising absolute or relative intensity values for each emotional category identifier, and the method comprises mapping each data set as a single representative or resultant two-dimensional vector into the circular reference frame, the vector representing the overall emotional profile and intensity of the text segment.

18. The method according to claim 1 wherein identifying text regions comprises identifying reference text segments that represent emotional transitions in text based on predefined emotional-change threshold or thresholds, and defining the text regions relative to those reference text segments.

19. The method according to claim 1 wherein identifying text regions comprises defining all text segments as an initial set of text regions, and iteratively grouping adjacent text segments into a set of larger final text regions based on analysis of the segment emotional data profiles and/or other predefined rules.

20. The method according to claim 1 wherein identifying text regions comprises:

(i) defining a first set of text regions of one or multiple adjacent text segments by grouping text segments with neutral segment emotional data profiles with adjacent non-neutral segment emotional data profiles;

(ii) reducing the first set of text regions into a smaller second set of text regions based on grouping at least some adjacent text regions from the first set into larger text regions until all the text regions in the second set reach a predetermined size threshold;

(iii) generating or determining an interim text region emotional data profile for each of the larger text regions in the second set based on the individual segment emotional data profiles of the segments within the region;

(iv) analysing the interim text region emotional data profiles of the second set of text regions and further grouping adjacent text regions having an emotional association based on an emotional-transition threshold or thresholds to generate a final set of text regions; and

(v) generating or determining a text region emotional data profile for each of the text regions in the final set based on the individual segment emotional data profiles of the text segments within the region and/or the interim text region emotional data profiles.

21. The method according to claim 20 wherein analysing the interim text region emotional data profiles of the second set of text regions and further grouping adjacent text regions having an emotional association based on an emotional-transition threshold to generate a final set of text regions comprises:

(i) mapping each interim text region emotional data profile or masked interim text region emotional data profile as a two-dimensional vector into the 2-dimensional circular reference frame of the continuous emotion model;

(ii) identifying emotional transitions between adjacent interim text regions based on analysing or comparing the distance from the origin and/or angular movement between two-dimensional vectors of adjacent interim text regions relative to emotional-transition thresholds; and

(iii) grouping of adjacent interim text regions based on the identified emotional transitions to generate a final set of text regions.

22. The method according to claim 20 wherein generating or determining a text region emotional data profile for each of the text regions in the final set comprises any one or more of the following:

aggregating or summing the individual segment emotional data profiles of the text segments of the text region or combining or summing the individual interim text region emotional data profiles of the text region,

selecting a prevailing or dominant segment emotional data profile or interim text region emotional data profile to represent the final text region, and/or

generating a representative text region emotional data profile based on statistical processing of the segment emotional data profiles or interim text region emotional data profiles.

23. The method according to claim 1 wherein the step of processing an accessible audio database or databases comprising audio data files and associated audio profile information to select an audio data file for playback in each audio region comprises translating the region emotional profiles of the text regions into a format or data set that is comparable to the audio profile information of the audio data files in the audio database or databases.

24. The method according to claim 1 wherein the step of processing an accessible audio database or databases comprising audio data files and associated audio profile information to select an audio data file for playback in each audio region comprises translating the audio profile information of the audio data files is into a format or data set that is comparable to the region emotional profiles.

25. The method according to claim 1 wherein the step of processing an accessible audio database or databases comprising audio data files and associated audio profile information to select an audio data file for playback in each audio region comprises translating both the region emotional profiles and audio profile information into a common format or data set so they can be compared for a match.

26. The method according to claim 1 wherein the step of processing an accessible audio database or databases comprising audio data files and associated audio profile information to select an audio data file for playback in each audio region comprises processing an accessible audio database comprising audio data files having associated audio profile information that is in the same format or is at least partly directly comparable to the format of the region emotional profiles.

27. The method according to claim 1 wherein the method further comprises applying semantic analysis to the processed text based on one or more thematic category identifiers having associated thematic lexicons to generate segment thematic profiles for the text segments.

28. The method according to claim 27 wherein the method comprises:

identifying a series of text regions each comprising a text segment or plurality of text segments having an emotional association and/or thematic association based on processing the segment emotional and thematic data profiles with respect to predefined rules; and

generating, for each identified text region and its associated audio region, a region emotional and thematic data profile based on the emotional and thematic profiles of the text segments within the text region.

29. The method according to claim 28 wherein the method comprises processing the accessible audio database or databases to select an audio data file for playback in each audio region based at least partly on the audio profile information of an audio data file corresponding or matching closely to the region emotional and thematic profile of the audio region.

30. The method according to claim 1 wherein the method further comprises processing the identified text regions to generate a thematic data profile for each region based on one or more thematic category identifiers having associated thematic lexicons, and generating a region emotional and thematic data profile for each text region and its associated audio region.

31. The method according to claim 30 wherein the method comprises processing the accessible audio database or databases to select an audio data file for playback in each audio region based at least partly on the audio profile information of an audio data file corresponding or matching closely to the region emotional and thematic profile of the audio region.

32. The method according to claim 1 wherein the soundtrack comprises a soundtrack timeline along which the one or more audio regions are configured for synchronised playback with corresponding text regions in the text as it is being read, each audio region having a position along the soundtrack timeline that is dynamically adjustable to maintain synchronization of the audio regions of the soundtrack with their respective text regions in the text as they are being read based on a reading speed variable indicative of the reader'"'"'s reading speed and/or a reading position variable indicative of the reader'"'"'s reading position in the text.

33. A soundtrack or soundtrack data generated according to the method of claim 1.

34. A system comprising a processor or processors configured to implement the method of claim 1.

35. A non-transitory computer-readable medium having stored thereon computer readable instructions that, when executed on a processing device or devices, cause the processing device to perform the method of claim 1.

36. A method of automatically generating a digital soundtrack intended for synchronised playback with associated speech audio, the method executed by a processing device or devices having associated memory, the method comprising:

syntactically and/or semantically analysing text representing or corresponding to the speech audio at a text segment level to generate an emotional profile for each text segment at least partly based on scanning the word or word concepts in the text segment relative to a predefined number of emotional category lexicons;

analysing the emotional profiles of the text segments to group those having an emotional association into larger text regions of multiple adjacent text segments; and

generating a soundtrack for the speech audio comprising audio regions that are configured for playback during the corresponding speech regions of the speech audio, and wherein the audio configured or selected for playback in the audio regions is based on or is a function of the emotional profiles of the group of text segments within the respective associated speech regions.

37. A method of automatically generating a digital soundtrack intended for synchronised playback with associated speech audio, the method executed by a processing device or devices having associated memory, the method comprising:

identifying emotional-transition points in text representing or corresponding to the speech audio based on syntactically and/or semantically analysing the text in the context of a continuous emotion model;

defining a series of audio regions for the soundtrack based on the identified emotional-transition points, each audio region corresponding to an associated speech region in the speech audio;

selecting an audio file or files for playback in each audio region based on an emotional profile determined for its associated speech region; and

generating a soundtrack for the speech audio based on the configured audio regions and selected audio files.