Identifying corresponding regions of content

• US 9,799,336 B2
• Filed: 08/03/2015
• Issued: 10/24/2017
• Est. Priority Date: 08/02/2012
• Status: Active Grant

• Current Assignee: Audible Incorporated (Amazon.com, Inc.)
• Original Assignee: Audible Incorporated (Amazon.com, Inc.)
• Inventors: Dzik, Steven C., Story, Jr., Guy A.
• Primary Examiner(s): AZAD, ABUL K
• Application Number:
  US14/817,148
• Publication Number:
  US 20150340038A1
• Time in Patent Office: 813 Days
• Field of Search: 704235, 704260, 704257, 704270
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A content alignment service may generate content synchronization information to facilitate the synchronous presentation of audio content and textual content. In some embodiments, a region of the textual content whose correspondence to the audio content is uncertain may be analyzed to determine whether the region of textual content corresponds to one or more words that are audibly presented in the audio content, or whether the region of textual content is a mismatch with respect to the audio content. In some embodiments, words in the textual content that correspond to words in the audio content are synchronously presented, while mismatched words in the textual content may be skipped to maintain synchronous presentation. Accordingly, in one example application, an audiobook is synchronized with an electronic book, so that as the electronic book is displayed, corresponding words of the audiobook are audibly presented.

Claims

1. A computer-implemented method comprising:

as implemented by one or more computing devices configured with specific computer-executable instructions,

obtaining an audio representation of a first content item, wherein the first content item includes at least one of audio content, text content, or video content;

obtaining a textual representation of a second content item, wherein the second content item includes at least one of audio content, text content, or video content, and wherein the second content item is distinct from the first content item;

identifying audio data of the audio representation that is preliminarily correlated to a region of the textual representation, wherein the audio representation is being paired with the textual representation for synchronous presentation with the textual representation;

determining that text within the region of the textual representation does not correspond to the audio data of the audio representation;

transcribing the audio data with automated speech recognition to generate a textual transcription of the audio data;

determining that the textual transcription corresponds to the text within the region of the textual representation; and

generating synchronization information that associates the audio data of the audio representation with the text within the region of the textual representation.

2. The computer-implemented method of claim 1, wherein determining that the text within the region of the textual representation does not correspond to the audio data of the audio representation comprises identifying at least one word within the text not represented in the audio data.

3. The computer-implemented method of claim 1, wherein transcribing the audio data with automated speech recognition to generate a textual transcription of the audio data comprises generating a language model using the text within the region of the textual representation and applying the language model to the audio data.

4. The computer-implemented method of claim 1, wherein determining that the textual transcription corresponds to the text within the region of the textual representation comprises:

identifying a word represented in both the textual transcription and the text within the region of the textual representation;

assigning a score value to the word; and

determining that the score value of the word satisfies a threshold value.

5. The computer-implemented method of claim 4, wherein the score value of the word indicates a frequency of occurrence of the word within the textual representation.

6. The computer-implemented method of claim 1, wherein determining that the textual transcription corresponds to the text within the region of the textual representation comprises:

generating a first phoneme string from the text within the region of the textual representation;

generating a second phoneme string from the textual transcription of the audio data; and

determining that the first phoneme string corresponds to the second phoneme string.

7. The computer-implemented method of claim 6, wherein determining that the first phoneme string corresponds to the second phoneme string comprises determining that a Levenshtein distance between the first phoneme string and the second phoneme string satisfies a threshold value.

8. The computer-implemented method of claim 6, wherein determining that the first phoneme string corresponds to the second phoneme string comprises determining that an acoustically confusable hypothesis for the first phoneme string corresponds to the second phoneme string.

9. A system comprising:

a non-transitory data store storing audio data of an audio representation of a first content item that includes at least one of audio content, text content, or video content, and text of a textual representation of a second content item that includes at least one of audio content, text content, or video content, wherein the first content item is distinct from the second content item, and wherein the audio representation is being paired with the textual representation for synchronous presentation with the textual representation; and

a processor in communication with the non-transitory data store and configured with specific computer-executable instructions that, when executed by the processor, cause the processor to at least;

identify a preliminary correlation between the audio data and a region of the textual representation;

determine that text within the region of the textual representation does not correspond to the audio data of the audio representation;

transcribe the audio data with automated speech recognition to generate a textual transcription of the audio data;

determine that the textual transcription corresponds to the text within the region of the textual representation; and

generate synchronization information that associates the audio data of the audio representation with the text within the region of the textual representation.

10. The system of claim 9, wherein the specific computer-executable instructions cause the processor to at least determine that the text within the region of the textual representation does not correspond to the audio data of the audio representation at least partly by identifying at least one word within the text not represented in the audio data.

11. The system of claim 9, wherein the specific computer-executable instructions cause the processor to at least transcribe the audio data according to an automated speech recognition routine at least partly by generating a language model using the text within the region of the textual representation and applying the language model to the audio data.

12. The system of claim 9, wherein the specific computer-executable instructions further cause the processor to at least:

identify a word represented in both the textual transcription and the text within the region of the textual representation; and

assign a score value to the word based at least in part on a commonality of the word;

wherein the specific computer-executable instructions cause the processor to determine that the textual transcription corresponds to the text within the region of the textual representation at least in part by determining that the score value of the word satisfies a threshold value.

13. The system of claim 12, wherein the commonality of the word is determined according to a predetermined data set assigning commonalties to a plurality of words.

14. The system of claim 9, wherein the specific computer-executable instructions further cause the processor to at least:

generate a first phoneme string from the text within the region of the textual representation; and

generate a second phoneme string from the textual transcription of the audio data;

wherein the specific computer-executable instructions cause the processor to determine that the textual transcription corresponds to the text within the region of the textual representation at least in part by determining that the first phoneme string corresponds to the second phoneme string.

15. The system of claim 14, wherein the first phoneme string corresponds to the second phoneme string when a Levenshtein distance between the first phoneme string and the second phoneme string satisfies a threshold value.

16. The system of claim 14, wherein the first phoneme string corresponds to the second phoneme string when an acoustically confusable hypothesis for the first phoneme string corresponds to an acoustically confusable hypothesis for the second phoneme string.

17. Non-transitory computer-readable storage media comprising computer-executable instructions, that, when executed by a computing system, cause the computing system to at least:

a non-transitory data store storing audio data of an audio representation of a first content item that includes at least one of audio content, text content, or video content, and text of a textual representation of a second content item that includes at least one of audio content, text content, or video content, wherein the first content item is distinct from the second content item, and wherein the audio representation is being paired with the textual representation for synchronous presentation with the textual representation; and

a processor in communication with the non-transitory data store and configured with specific computer-executable instructions that, when executed by the processor, cause the processor to at least;

identify a preliminary correlation between the audio data and a region of the textual representation;

determine that text within the region of the textual representation does not correspond to the audio data of the audio representation;

transcribe the audio data with automated speech recognition to generate a textual transcription of the audio data;

determine that the textual transcription corresponds to the text within the region of the textual representation; and

generate synchronization information that associates the audio data of the audio representation with the text within the region of the textual representation.

18. The non-transitory computer-readable storage media of claim 17, wherein the computer-executable instructions cause the computing system to at least determine that the text within the region of the textual representation does not correspond to the audio data of the audio representation at least partly by identifying at least one word within the text not represented in the audio data.

19. The non-transitory computer-readable storage media of claim 17, wherein the computer-executable instructions cause the computing system to at least apply automated speech recognition to the audio data to generate a textual transcription of the audio data at least partly by generating a language model using the text within the region of the textual representation and applying the language model to the audio data.

20. The non-transitory computer-readable storage media of claim 17, wherein the computer-executable instructions further cause the computing system to at least:

identify a word represented in both the textual transcription and the text within the region of the textual representation; and

assign a score value to the word;

wherein the computer-executable instructions cause the computing system to determine that the textual transcription corresponds to the text within the region of the textual representation at least in part by determining that the score value of the word satisfies a threshold value.