Interactive eReader interface generation based on synchronization of textual and audial descriptors

• US 10,671,251 B2
• Filed: 12/22/2017
• Issued: 06/02/2020
• Est. Priority Date: 12/22/2017
• Status: Active Grant

• Current Assignee: Fathom Technologies LLC
• Original Assignee: Arbordale Publishing, LLC
• Inventors: German, Lee B.
• Primary Examiner(s): Al Kawsar, Abdullah
• Assistant Examiner(s): Nguyen, Kenny
• Application Number:
  US15/852,350
• Publication Number:
  US 20190196675A1
• Time in Patent Office: 893 Days
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems

Abstract

The present invention is directed to systems and methods for providing an improved interactive and educational eBook platform through an improved eReader. The system provides a platform through which a book is transformed into an interactive, multi-language, assisted reading, read-aloud eBook and is displayed in an eReader with an improved graphical user interface that provides features which enhance the effectiveness of eBook learning.

Claims

1. A system for transforming eBooks and providing an improved eReader interface, comprising:

a processor coupled with memory and at least one database, wherein the processor is configured to;

convert a digital book into image files and extract words, characters, and punctuation marks from the image files;

generate textual descriptors, including at least a page number, a word or character length, and a language, for each of the extracted words, characters, and punctuation marks;

store the extracted words, characters, and punctuation marks with the textual descriptors associated with the extracted words, characters, and punctuation marks in the database;

retrieve an audio file for the corresponding digital book and identify timestamps of the audio file that correspond to specific words or characters;

in accordance with the identified timestamps, apply keyframes at a beginning and an end of each word and segment the audio file into audio segments based on the keyframes;

generate audial descriptors, including at least the keyframes, a corresponding word, audial runtime of the corresponding word, and a file size, for each audio segment;

store the audio segments with their associated audial descriptors in the database;

use a synchronization engine to pair the extracted words or characters;

with the audio segments, wherein pairing the extracted words or characters with the audio segments includes;

matching a sequence of the extracted words or characters with the associated textual descriptors stored in the database with a sequence of the audial descriptors for the audio segments stored in the database;

aggregating a sequence of the matched textual descriptors and audial descriptors into synchronization data; and

inserting the extracted punctuation marks into the aggregated sequence to be part of the synchronization data and outputting the synchronization data to a HyperText Markup Language (HTML) Generator;

use the HTML Generator to transform the output of the synchronization engine into eReader-displayable content by embedding the output into tags;

wherein embedding the output into tags includes outputting electronic markup, stylesheet, and/or semi-structured data for the extracted words, characters, punctuation marks, and the corresponding audio segments based on the synchronized textual descriptors and audial descriptors from the output of the synchronization engine; and

a graphical user interface (GUI), wherein the GUI is configured to;

display the electronic markup, stylesheet, and/or semi-structured data on a human-machine interface (HMI);

highlight each of the words or the characters for a time based on the audial descriptors of the corresponding audio segments, including the keyframes at the beginning and the end of each word;

adjust playback speed of an audio file based on a selection input; and

receive a word or character selection input, modify a word or character highlight based on the word or character selection input, and initiate playback of the corresponding audio segments according to the word selection input and the synchronized textual descriptors and audial descriptors for the word or character from the output of the synchronization engine.

2. The system of claim 1, wherein the textual descriptors further include definitions, translations, and a number of occurrences, and wherein the audial descriptors further include corresponding page numbers, corresponding audio files or text files, and related audio clips.

3. The system of claim 1, wherein the GUI is further configured to receive an indication of a digital book selection, load a corresponding digital book from the at least one database, and display the corresponding digital book.

4. The system of claim 1, wherein the extraction of the words, the characters, and the punctuation marks deletes the words, the characters, and the punctuation marks from the digital book.

5. The system of claim 1, wherein the GUI includes a dynamic text container, which is configured to display the electronic markup, stylesheet, and/or semi-structured data according to the textual descriptors, including a font size and typeface, wherein dimensions of the dynamic text container are preset, and wherein the dynamic text container is further configured to enable scrolling for overflow text within the dynamic text container.

6. The system of claim 1, wherein the processor is further configured to output language-specific characters and special characters in markup-specific formats.

7. The system of claim 1, wherein the GUI is further configured to receive a graphical selection from the HMI, and wherein the GUI is further configured to initiate playback of an audio clip related to the graphical selection or a portion of an audio clip related to the graphical selection.

8. The system of claim 1, wherein the GUI is further configured to track usage data relating to the digital book, store the usage data in a usage database, display the usage data in an expandable chart, and automatically populate the expandable chart with read digital books, wherein the expandable chart includes an expandable timeline and expandable categories and wherein the expandable chart is presented in tabular form.

9. The system of claim 1, wherein the words, the characters, and the punctuation marks include a first language set of words, characters, and punctuation marks, wherein the textual descriptors further include a corresponding first language, and wherein upon receiving an indication of interaction with the GUI, the processor is further configured to replace the first language set of words, characters, and punctuation marks with a second language set of words, characters, and punctuation marks and corresponding second language textual descriptors.

10. The system of claim 9:

wherein a dynamic text container is configured to display the second language set of words, characters, and punctuation marks according to the second language textual descriptors;

wherein dimensions of the dynamic text container are equal preset values for both the first language set of words, characters, and punctuation marks and the second language set of words, characters, and punctuation marks; and

wherein the dynamic text container is further configured to enable scrolling for overflow text within the dynamic text container.

11. The system of claim 1, wherein the GUI is further configured to receive a syllable selection input via the GUI and highlight a syllable according to the syllable selection input.