SWITCHING BETWEEN TEXT DATA AND AUDIO DATA BASED ON A MAPPING

• US 20120310649A1
• Filed: 10/06/2011
• Published: 12/06/2012
• Est. Priority Date: 06/03/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cannistraro, Alan C., Walsh, Raymond, Robbin, Gregory S., Dougherty, Casey M., Hajj, Melissa Breglio
• Granted Patent: US 10,672,399 B2
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Techniques are provided for creating a mapping that maps locations in audio data (e.g., an audio book) to corresponding locations in text data (e.g., an e-book). Techniques are provided for using a mapping between audio data and text data, whether the mapping is created automatically or manually. A mapping may be used for bookmark switching where a bookmark established in one version of a digital work (e.g., e-book) is used to identify a corresponding location with another version of the digital work (e.g., an audio book). Alternatively, the mapping may be used to play audio that corresponds to text selected by a user. Alternatively, the mapping may be used to automatically highlight text in response to audio that corresponds to the text being played. Alternatively, the mapping may be used to determine where an annotation created in one media context (e.g., audio) will be consumed in another media context.

Claims

1. A method comprising:

obtaining location data that indicates a specified location within a textual version of a work;

inspecting a mapping between a plurality of audio locations in an audio version of the work and a corresponding plurality of text locations in the textual version of the work to;

determine a particular text location, of the plurality of text locations, that corresponds to the specified location, and

based on the particular text location, determine a particular audio location, of the plurality of audio locations, that corresponds to the particular text location;

providing the particular audio location, that was determined based on the particular text location, to a media player to cause the media player to establish the particular audio location as a current playback position of the audio data;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein:

obtaining comprises a server receiving, over a network, the location data from a first device;

inspecting and providing are performed by the server;

providing comprises the server sending the particular audio location to a second device that executes the media player.

3. The method of claim 2, wherein the second device and the first device are the same device.

4. The method of claim 1 wherein obtaining, inspecting, and providing are performed by a computing device that is configured to display the textual version of the work and that executes the media player.

5. The method of claim 1, further comprising determining, at a device that is configured to display the textual version of the work, the location data without input from a user of the device.

6. The method of claim 1, further comprising:

receiving input from a user; and

in response to receiving the input, determining the location data based on the input.

7. The method of claim 6, wherein:

providing the particular audio location to the media player comprises providing the particular audio location to the media player to cause the media player to process the audio data beginning at the current playback position, which causes the media player to generate audio from the processed audio data; and

causing the media player to process the audio data is performed in response to receiving the input.

8. The method of claim 7, wherein:

the input selects multiple words in the textual version of the work;

the specified location is a first specified location;

the location data also indicates a second specified location, within the textual version of the work, that is different than the first specified location;

inspecting further comprises inspecting the mapping to;

determine a second particular text location, of the plurality of text locations, that corresponds to the second specified location, and

based on the second particular text location, determine a second particular audio location, of the plurality of audio locations, that corresponds to the second particular text location; and

providing the particular audio location to the media player comprises providing the second particular audio location to the media player to cause the media player to cease processing the audio data when the current playback position arrives at or near the second particular audio location.

21. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 1.

22. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 2.

23. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 3.

24. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 4.

25. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 5.

26. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 6.

27. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 7.

28. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 8.

9. A method comprising:

while a first version of a work is processed, obtaining annotation data that is based on input from a user;

storing association data that associates the annotation data with the work; and

while a second version of the work is processed, causing information about the annotation data to be displayed, wherein the second version is different than the first version;

wherein the method is performed by one or more computing devices.

10. The method of claim 9, wherein:

obtaining comprises determining location data that indicates a specified location within the first version of the work;

storing comprises storing the location data in association with the work;

the specified location corresponds to a particular location within the second version of the work; and

causing comprises causing the information about the annotation data to be displayed in association with the particular location in the second version.

11. The method of claim 10, wherein:

the first version is an audio version of the work and the second version is a textual version of the work;

causing information about the annotation data to be displayed comprises;

determining when a portion of the textual version of the work that corresponds to the particular location is displayed; and

in response to determining that a portion of the textual version of the work that corresponds to the particular location is displayed, causing information about the annotation data to be displayed.

12. The method of claim 10, wherein:

the first version is a textual version of the work and the second version is an audio version of the work;

causing information about the annotation data to be displayed comprises;

determining when a portion of the audio version of the work that corresponds to the particular location is played; and

in response to determining that a portion of the textual version of the work that corresponds to the particular location is played, causing information about the annotation data to be displayed.

13. The method of claim 9, wherein:

the annotation data includes text data; and

causing information about the annotation data to be displayed comprises causing the text data to be displayed.

14. The method of claim 9, wherein:

the annotation data includes voice data; and

causing information about the annotation data to be displayed comprises causing the voice data to be processed to generate audio.

29. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 9.

30. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 10.

31. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 11.

32. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 12.

33. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 13.

34. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 14.

15. A method comprising:

receiving data that establishes a first bookmark within a first version of a work;

inspecting a mapping between a plurality of first locations in the first version of the work and a corresponding plurality of second locations in a second version of the work to;

determine a particular first location, of the plurality of first locations, that corresponds to the first bookmark, and

based on the particular first location, determine a particular second location, of the plurality of second locations, that corresponds to the particular first location;

wherein the first version of the work is different than the second version of the work;

causing data that establishes the particular second location as a second bookmark within the second version of the work to be stored;

wherein the method is performed by one or more computing devices.

16. The method of claim 15, wherein:

receiving comprises a server receiving, over a network, the input from a first device;

inspecting is performed by the server;

causing comprises the server sending the particular second location to a second device.

17. The method of claim 16, wherein the first device and the second device are different devices.

18. The method of claim 15, wherein the first version of the work is one of an audio version of the work or a text version of the work and the second version of the work is the other of the audio version or the text version.

35. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 15.

36. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 16.

37. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 17.

38. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 18.

19. A method comprising:

causing a portion of text of a work to be displayed by a device;

while the portion of text is displayed;

receiving, at the device, audio input from a user;

in response to receiving the audio input;

analyzing the audio input to identify one or more words;

determining whether the one or more words are reflected in the portion of the text;

in response to determining that the one or more words are reflected in the portion of the text, causing a visual indication to be displayed by the device.

20. The method of claim 19, wherein causing the visual indication to be displayed comprises causing text data that corresponds to the one or more words to be highlighted.

39. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 19.

40. One or more storage media storing instructions which, when executed by one or more processors, causes performance of the method recited in claim 20.