IDENTIFYING CORRESPONDING REGIONS OF CONTENT

• US 20140039887A1
• Filed: 09/05/2012
• Published: 02/06/2014
• Est. Priority Date: 08/02/2012
• Status: Active Grant

• Current Assignee: Audible Incorporated (Amazon.com, Inc.)
• Original Assignee: Guy A. Story, Jr.
• Inventors: Dzik, Steven C., Story, Guy A. Jr.
• Granted Patent: US 9,099,089 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A content alignment service may generate content synchronization information to facilitate the synchronous presentation of audio content and textual content. In some embodiments, a region of the textual content whose correspondence to the audio content is uncertain may be analyzed to determine whether the region of textual content corresponds to one or more words that are audibly presented in the audio content, or whether the region of textual content is a mismatch with respect to the audio content. In some embodiments, words in the textual content that correspond to words in the audio content are synchronously presented, while mismatched words in the textual content may be skipped to maintain synchronous presentation. Accordingly, in one example application, an audiobook is synchronized with an electronic book, so that as the electronic book is displayed, corresponding words of the audiobook are audibly presented.

Claims

1. A system comprising:

an electronic data store configured to store;

an audiobook; and

an electronic book that is a companion to the audiobook;

a computing device in communication with the electronic data store, the computing device configured to;

generate a textual transcription of the audiobook;

identify, with respect to the textual transcription, an uncertain region in the electronic book;

identify a region of the audiobook that is preliminarily aligned to the uncertain region;

apply a language model to the region of the audiobook to generate an updated textual transcription;

determine that a portion of the updated textual transcription substantially corresponds to a portion of the electronic book including the uncertain region; and

generate content synchronization information, wherein the content synchronization information facilitates the synchronous presentation of the portion of the electronic book including the uncertain region and a portion of the audiobook from which the substantially corresponding portion of the updated textual transcription was generated.

2. The system of claim 1, wherein the computing device is further configured to provide the content synchronization information to a separate computing device.

3. The system of claim 1, wherein the computing device is further configured to synchronously present the portion of the electronic book including the uncertain region and the portion of the audiobook from which the substantially corresponding portion of the updated textual transcription was generated.

4. The system of claim 1, wherein the language model is based at least in part on one or more words of the uncertain region.

5. A computer-implemented method comprising:

as implemented by one or more computing devices configured with specific computer-executable instructions,

identifying an uncertain region in an item of textual content;

identifying a region of a companion item of audio content that is preliminarily aligned to the uncertain region;

generating a textual transcription of the region of the item of audio content;

determining that a portion of the textual transcription substantially corresponds to a portion of the item of textual content to which the uncertain region belongs; and

generating content synchronization information for synchronizing presentation of the portion of the item of textual content and a portion of the item of audio content from which the substantially corresponding portion of the updated textual transcription was generated.

6. The computer-implemented method of claim 5, wherein the portion of the textual transcription substantially corresponds to the portion of the item of textual content if they have a portion score satisfying a threshold.

7. The computer-implemented method of claim 5, wherein the portion of the textual transcription substantially corresponds to the portion of the item of textual content if at least a threshold percentage of words of the portion of the textual transcription correspond to words of the portion of the item of textual content.

8. The computer-implemented method of claim 7, wherein a word of the portion of the textual transcription corresponds to a word of the portion of the item of textual content if the word of the portion of the textual transcription substantially matches and chronologically corresponds to the word of the portion of the item of textual content.

9. The computer-implemented method of claim 5, wherein:

the textual transcription is generated based at least in part on a language model; and

the language model is based at least in part on one or more words belonging to the uncertain region.

10. The computer-implemented method of claim 5, wherein the uncertain region of the item of textual content is identified based at least in part by comparing the uncertain region to an initial transcription of the item of audio content.

11. A system for synchronizing presentation of an item of audio content to a companion item of textual content, the system comprising:

an electronic data store configured to store content synchronization information; and

a computing device in communication with the electronic data store, the computing device being configured to;

identify, with respect to a textual transcription of the item of audio content, an uncertain region in the companion item of textual content, the uncertain region comprising one or more words;

identify a region of the textual transcription that is preliminarily aligned to the uncertain region, the region of the textual transcription comprising one or more words;

convert the one or more words of the uncertain region to a first phoneme string;

convert the one or more words of the region of the textual transcription to a second phoneme string; and

if the first phoneme string substantially corresponds to the second phoneme string, generate content synchronization information that facilitates the synchronous presentation of the one or more words of the uncertain region in the companion item of textual content and the one or more words of the item of audio content from which the region of the textual transcription was generated.

12. The system of claim 11, wherein the first phoneme string substantially corresponds to the second phoneme string if the first phoneme string is within a threshold Levenshtein distance from the second phoneme string.

13. The system of claim 11, wherein:

the computing device is further configured to generate an acoustically confusable hypothesis for the first phoneme string; and

the first phoneme string substantially corresponds to the second phoneme string if the acoustically confusable hypothesis for the first phoneme string is at least substantially similar to the second phoneme string.

14. The system of claim 11, wherein:

the computing device is further configured to generate an acoustically confusable hypothesis for the second phoneme string; and

the first phoneme string substantially corresponds to the second phoneme string if the acoustically confusable hypothesis for the second phoneme string is at least substantially similar to the first phoneme string.

15. The system of claim 11, wherein:

the computing device is further configured to generate a first acoustically confusable hypothesis for the first phoneme string and a second acoustically confusable hypothesis for the second phoneme string; and

the first phoneme string substantially corresponds to the second phoneme string if the first acoustically confusable hypothesis is at least substantially similar to the second acoustically confusable hypothesis.

16. A non-transitory computer-readable medium having a computer-executable module, the computer-executable module configured to:

identify an uncertain region in an item of textual content, the uncertain region comprising one or more words;

identify a region of the item of audio content that is preliminarily aligned to the uncertain region;

generate a textual transcription of the region of the item of audio content, the textual transcription comprising one or more words;

determine whether the one or more words of the uncertain region substantially correspond to the one or more words of the textual transcription of the preliminarily aligned region of the item of audio content; and

if the one or more words of the uncertain region substantially correspond to the one or more words of the updated textual transcription of the preliminarily aligned region of audio content, generate content synchronization information;

wherein the content synchronization information facilitates the synchronous presentation of the uncertain region in the item of textual content and the preliminarily aligned region of the item of audio content.

17. The non-transitory computer-readable medium of claim 16, wherein the language model is based at least in part on the one or more words of the uncertain region.

18. The non-transitory computer-readable medium of claim 16, wherein the computer-executable module is further configured to:

if the one or more words of the uncertain region do not substantially correspond to the one or more words of the textual transcription of the region of the item of audio content,

convert the one or more words of the uncertain region to a first phoneme string;

convert the one or more words of the updated textual transcription to a second phoneme string;

determine whether the first phoneme string substantially corresponds to the second phoneme string; and

if the first phoneme string substantially corresponds to the second phoneme string, generate content synchronization information;

wherein the content synchronization information facilitates the synchronous presentation of the uncertain region and the preliminarily aligned region of the item of audio content.

19. The non-transitory computer-readable medium of claim 18, wherein the first phoneme string substantially corresponds to the second phoneme string if the first phoneme string is within a threshold Levenshtein distance from the second phoneme string.

20. The non-transitory computer-readable medium of claim 18, wherein:

the computer-executable module is further configured to generate an acoustically confusable hypothesis for the first phoneme string; and

the first phoneme string substantially corresponds to the second phoneme string if the acoustically confusable hypothesis for the first phoneme string is at least substantially similar to the second phoneme string.

21. The non-transitory computer-readable medium of claim 18, wherein:

the computer-executable module is further configured to generate an acoustically confusable hypothesis for the second phoneme string; and

the first phoneme string substantially corresponds to the second phoneme string if the acoustically confusable hypothesis for the second phoneme string is at least substantially similar to the first phoneme string.

22. The non-transitory computer-readable medium of claim 18, wherein:

the computer-executable module is further configured to generate a first acoustically confusable hypothesis for the first phoneme string and a second acoustically confusable hypothesis for the second phoneme string; and

the first phoneme string substantially corresponds to the second phoneme string if the first acoustically confusable hypothesis is at least substantially similar to the second acoustically confusable hypothesis.

23. A computer-implemented method comprising:

as implemented by one or more computing devices configured with specific computer-executable instructions,

identifying an uncertain region in an item of textual content;

identifying a region of the item of audio content that is preliminarily aligned to the uncertain region;

generating a textual transcription of the region of the item of audio content;

identifying a significant corresponding word, wherein the significant corresponding word appears in both the textual transcription and the item of textual content; and

generating content synchronization information, wherein the content synchronization information facilitates the synchronous presentation of the significant corresponding word in both the item of textual content and the item of audio content.

24. The computer-implemented method of claim 23, wherein the uncertain region comprises at least a threshold number of words.

25. The computer-implemented method of claim 23, wherein the significant corresponding word has a word score that satisfies a threshold.

26. The computer-implemented method of claim 25, wherein the word score of the significant corresponding word is based at least in part on at least one of:

a number of letters included in the significant corresponding word;

a frequency of one or more letters included in the significant corresponding word; and

a number of syllables included in the significant corresponding word.

27. The computer-implemented method of claim 23 further comprising:

identifying a substantial acoustic similarity between a first word string of the item of textual content and a second word string of the textual transcription; and

identifying a subregion of the item of audio content that corresponds to the first word string;

wherein the first word string occurs substantially within the uncertain region in the item of textual content; and

wherein the content synchronization information further facilitates the synchronous presentation of the first word string in the item of textual content and the corresponding subregion of the item of audio content.

28. The computer-implemented method of claim 27, wherein neither the first word string nor the second word string include the significant corresponding word.

29. The computer-implemented method of claim 27, wherein the first word string comprises at least a threshold number of words.