Aligning body matter across content formats

• US 10,109,278 B2
• Filed: 09/05/2012
• Issued: 10/23/2018
• Est. Priority Date: 08/02/2012
• Status: Active Grant

• Current Assignee: Audible Incorporated (Amazon.com, Inc.)
• Original Assignee: Audible Incorporated (Amazon.com, Inc.)
• Inventors: Dzik, Steven C., Story, Jr., Guy A.
• Primary Examiner(s): Padmanabhan, Kavita
• Assistant Examiner(s): El-Bkaily, Ahmad M
• Application Number:
  US13/604,486
• Publication Number:
  US 20140040713A1
• Time in Patent Office: 2,239 Days
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A content alignment service is described that may generate content synchronization information to facilitate the synchronous presentation of corresponding audio content and textual content. In some embodiments, portions of body text (as opposed to front matter, such as a table of contents; or back matter, such as an index) in the textual content are identified and synchronized with corresponding audio content. In one example application, an audiobook may be synchronized with an electronic book. As the body text portions of the electronic book are consumed, corresponding words of the audiobook may be audibly presented.

Claims

1. A system for aligning content, the system comprising:

an electronic data store configured to store;

an electronic book comprising;

a plurality of paragraphs of body text, and

matter other than body text, wherein the matter other than body text comprises text within at least front matter and back matter; and

an audiobook that is a companion to the electronic book; and

a physical computing device in communication with the electronic data store, the physical computing device configured to;

generate a textual transcription of the audiobook by applying a speech-to-text recognition routine on the audiobook;

identify a portion of the textual transcription that includes text also included in a paragraph of the electronic book;

determine a level of correlation between words in the paragraph of the electronic book and words in the portion of the textual transcription;

determine that the level of correlation satisfies a threshold value;

in response to determining that the level of correlation satisfies the threshold value, identify the paragraph of the electronic book as body text;

identify a first portion of the electronic book that does not satisfy the threshold value with respect to the textual transcription;

determine that the first portion of the electronic book that does not satisfy the threshold value is front matter based at least in part on a determination that the first portion of the electronic book that does not satisfy the threshold value appears within the electronic book prior to an earliest portion of the electronic book for which a corresponding portion of the audiobook is identified;

identify a second portion of the electronic book that does not satisfy the threshold value with respect to the textual transcription;

determine that the second portion of the electronic book that does not satisfy the threshold value is back matter based at least in part on a determination that the second portion of the electronic book that does not satisfy the threshold value appears within the electronic book after a last portion of the electronic book for which a corresponding portion of the audiobook is identified; and

generate content synchronization information that identifies (a) portions of the audiobook that correspond to the paragraphs of the body text and (b) further identifies the matter other than body text in the electronic book, wherein the content synchronization information indicates that the matter other than body text in the electronic book, including the first portion and second portion of the electronic book, does not correspond to any portion of the audiobook,

wherein the content synchronization information indicates that the paragraph, excluding the matter other than body text, should be presented in synchronization with a portion of the audiobook from which the corresponding portion of the textual transcription was generated.

2. The system of claim 1, wherein the physical computing device is further configured to provide the content synchronization information to a separate computing device.

3. The system of claim 1, wherein the physical computing device is further configured to synchronously present the paragraph of the electronic book and the portion of the audiobook from which the corresponding portion of the textual transcription was generated.

4. A computer-implemented method for aligning content, the computer-implemented method comprising:

as implemented by one or more computing devices configured with specific computer-executable instructions,

obtaining a textual transcription of an item of content comprising audio content;

identifying a portion of the textual transcription that includes text also included in a portion of a companion item of textual content, wherein the textual content includes body text and matter other than body text;

determining a level of correlation between words in the portion of the companion item of textual content and words in the portion of the textual transcription;

determining that the level of correlation satisfies a threshold value;

in response to determining that the level of correlation satisfies a threshold value, identifying the portion of the companion item of textual content as including body text;

identifying a second portion of the companion item of textual content that does not satisfy the threshold value with respect to any portion of the textual transcription;

determining that the second portion of the companion item of textual content that does not satisfy the threshold value is front matter based at least in part on a determination that the second portion of the companion item of textual content that does not satisfy the threshold value appears within the companion item of textual content prior to an earliest portion of the companion item of textual content for which a corresponding portion of the audio content is identified; and

generating content synchronization information that indicates (a) portions of the audio content that correspond to body text of the companion item of textual content and (b) further indicates that the matter other than body text in the textual content does not correspond to any portion of the audio content, wherein the matter other than body text includes the second portion of the companion item of textual content determined to be front matter,

wherein the content synchronization information indicates that the body text included in the portion of the companion item of textual content should be presented in synchronization with a portion of the audio content that corresponds to the body text included in the portion of the textual transcription.

5. The computer-implemented method of claim 4, wherein obtaining the textual transcription comprises generating the textual transcription from the audio content.

6. The computer-implemented method of claim 4, wherein determining the level of correlation between words in the portion of the companion item of textual content and words in the portion of the textual transcription comprises computing a correlation measure for a block of the companion item of textual content with respect to the textual transcription, the block comprising one or more portions of the companion item of textual content.

7. The computer-implemented method of claim 4, wherein the body text portion comprises at least one of a word, a phrase, a sentence, a paragraph, and a line of dialogue.

8. The computer-implemented method of claim 4, wherein the companion item of textual content is an electronic book.

9. The computer-implemented method of claim 4, wherein the item of content comprising audio content is an audiobook.

10. The computer-implemented method of claim 4, wherein the item of content comprising audio content further comprises video content.

17. The computer-implemented method of claim 4, further comprising determining the level of correlation between words in the portion of the companion item of textual content and words in the portion of the textual transcription from metadata associated with at least one of the companion item of textual content and the portion of the textual transcription.

20. The computer-implemented method of claim 6, wherein determining that the level of correlation satisfies a threshold value comprises determining that the correlation measure for the block to which a portion of the companion item of textual content belongs, satisfies the threshold value.

11. A system for aligning content, the system comprising:

an electronic data store configured to store;

a transcription of an item of content comprising audio content; and

a companion item of textual content, wherein the companion item of textual content comprises;

a plurality of paragraphs of body text, and

matter other than body text; and

a physical computing device in communication with the electronic data store, the physical computing device configured to;

identify, in the transcription, a portion of the transcription that includes text also included in a portion of the companion item of textual content;

determine a level of correlation between words in the portion of the companion item of textual content and words in the portion of the transcription;

determine that the level of correlation satisfies a threshold value;

in response to determining that the level of correlation satisfies a threshold value, identify the portion of the companion item of content as body text;

identify a second portion of the companion item of textual content that does not satisfy the threshold value with respect to any portion of the transcription;

determine that the second portion of the companion item of textual content that does not satisfy the threshold value is back matter based at least in part on a determination that the second portion of the companion item of textual content that does not satisfy the threshold value appears within the companion item of textual content after a last portion of the companion item of textual content for which a corresponding portion of the transcription is identified; and

generate content synchronization information that identifies (a) portions of the audio content that correspond to body text of the companion item of textual content and (b) further identifies the matter other than body text in the companion item, wherein the content synchronization information indicates that the matter other than body text in the companion item does not correspond to the audio content, wherein the matter other than body text includes the second portion of the companion item of textual content determined to be back matter,

wherein the content synchronization information indicates that the portion of the companion item of textual content, excluding the matter other than body text, should be presented in synchronization with a portion of the audio content that corresponds to the portion of the transcription.

12. The system of claim 11, wherein the level of correlation is determined based at least in part on at least one of metadata associated with the companion item of textual content and metadata associated with the portion of the transcription.

13. The system of claim 11, wherein the level of correlation is determined based at least in part on input received from a human interaction task system.

14. The system of claim 11, wherein the portion of the companion item of content identified as body text comprises at least one of a word, sentence, or paragraph.

15. The system of claim 11, wherein the threshold value comprises a threshold percentage of words in the portion of the transcription corresponding to words in the portion of the companion item of textual content.

16. The system of claim 15, wherein a word in the portion of the transcription corresponds to a word in the portion of the companion item of textual content when the word in the portion of the transcription substantially matches and chronologically corresponds to the word in the portion of the companion item of textual content.

18. The system of claim 11, wherein the physical computing device is further configured to determine the level of correlation between words in the portion of the companion item of textual content and words in the portion of the transcription by computing a correlation measure for a block of the companion item of textual content with respect to the transcription, the block comprising one or more portions of the companion item of textual content.

19. The system of claim 11, wherein the physical computing device is further configured to synchronously present the portion of the companion item of textual content and the portion of the audio content that corresponds to the portion of the transcription.

21. The system of claim 18, wherein the physical computing device configured to determine that the level of correlation satisfies a threshold value is configured to determine that the correlation measure for the block to which a portion of the companion item of textual content belongs, satisfies the threshold value.