Clarifying audible verbal information in video content

• US 10,206,014 B2
• Filed: 09/16/2014
• Issued: 02/12/2019
• Est. Priority Date: 06/20/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Trollope, Ingrid McAulay, Oztaskent, Ant, Volovich, Yaroslav
• Primary Examiner(s): Phantana-angkool, David
• Application Number:
  US14/488,213
• Publication Number:
  US 20150373428A1
• Time in Patent Office: 1,610 Days
• Field of Search: 715719
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 40/00 Handling natural language data speech analysis or synthesis, speech recognition G10L
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

A method at a server includes: receiving a user request to clarify audible verbal information associated with a media content item playing in proximity to a client device, where the user request includes an audio sample of the media content item and a user query, and the audio sample corresponds to a portion of the media content item proximate in time to issuance of the user query; in response to the user request: identifying the media content item and a first playback position in the media content corresponding to the audio sample; in accordance with the first playback position and identity of the media content item, obtaining textual information corresponding to the user query for a respective portion of the media content item; and transmitting to the client device at least a portion of the textual information.

Claims

1. A method, comprising:

at a server;

receiving from a first client device a user request to clarify audible verbal information of a media content item playing on a second client device in proximity to the first client device,

wherein the user request includes an audio sample of the media content item and a user query issued to the first client device by a user, the user query regarding a subset, not all, of audio content of the media content item; and

in response to the user request;

identifying the media content item based on the audio sample;

identifying a portion of the media content containing the subset of audio content based on the audio sample and content of the user query;

in accordance with the identity of the media content item, obtaining textual information responsive to the user query for the portion of the media content item; and

transmitting to the first client device at least a portion of the textual information for presentation at the first client device.

2. The method of claim 1, wherein the textual information comprises song lyrics.

3. The method of claim 1, wherein the textual information comprises a transcription of speech.

4. The method of claim 1, wherein the audible verbal information is associated with a plurality of speakers;

and the user request comprises a request to clarify the audible verbal information with respect to a respective speaker.

5. The method of claim 1, wherein the textual information comprises subtitle data associated with the media content item.

6. The method of claim 1, wherein the textual information comprises information obtained from an online document.

7. The method of claim 1, further comprising:

receiving from the first client device a second user request to clarify audible verbal information associated with the media content item playing on the second media device in proximity to the first client device, wherein the second user request includes a second audio sample of the media content item and a second user query issued to the first client device by the user, wherein the second audio sample corresponds to a second portion of the audio content of the media content item output during the playback of the media content item by the second client device proximate in time to the issuance of the second user query and recorded by the first client device; and

in response to the second user request;

identifying a second playback position in the media content corresponding to the second audio sample;

in accordance with the second playback position and the identity of the media content item, obtaining second textual information corresponding to the second user query for a second portion of the media content item; and

transmitting to the first client device at least a portion of the second textual information for presentation at the first client device.

8. The method of claim 1, wherein the textual information comprises a translation of the audible verbal information corresponding to the respective portion of the media content item.

9. The method of claim 1, further comprising:

obtaining or generating a translation of the textual information; and

transmitting to the client device at least a portion of the translation, wherein the at least a portion of the translation corresponds to the at least a portion of the textual information.

10. The method of claim 1, wherein the media content item is live television content.

11. The method of claim 1, wherein the media content item is previously broadcast television content.

12. The method of claim 1, wherein the media content item is recorded content.

13. The method of claim 1, wherein the media content item is streaming content.

14. A server system, comprising:

memory;

one or more processors; and

one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions for;

receiving from a first client device a user request to clarify audible verbal information of a media content item playing on a second client device in proximity to the first client device,

wherein the user request includes an audio sample of the media content item and a user query issued to the first client device by a user, the user query regarding a subset, not all, of audio content of the media content item; and

in response to the user request;

identifying the media content item based on the audio sample;

identifying a portion of the media content containing the subset of audio content based on the audio sample and content of the user query;

in accordance with the identity of the media content item, obtaining textual information responsive to the user query for the portion of the media content item; and

transmitting to the first client device at least a portion of the textual information for presentation at the first client device.

15. The system of claim 14, wherein the audible verbal information is associated with a plurality of speakers;

and the user request comprises a request to clarify the audible verbal information with respect to a respective speaker.

16. The system of claim 14, further comprising instructions for:

receiving from the first client device a second user request to clarify audible verbal information associated with the media content item playing on the second media device in proximity to the first client device, wherein the second user request includes a second audio sample of the media content item and a second user query issued to the first client device by the user, wherein the second audio sample corresponds to a second portion of the audio content of the media content item output during the playback of the media content item by the second client device proximate in time to the issuance of the second user query and recorded by the first client device; and

in response to the second user request;

identifying a second playback position in the media content corresponding to the second audio sample;

in accordance with the second playback position and the identity of the media content item, obtaining second textual information corresponding to the second user query for a second portion of the media content item; and

transmitting to the first client device at least a portion of the second textual information for presentation at the first client device.

17. The system of claim 14, further comprising instructions for:

obtaining or generating a translation of the textual information; and

transmitting to the client device at least a portion of the translation, wherein the at least a portion of the translation corresponds to the at least a portion of the textual information.

18. A non-transitory computer readable storage medium storing one or more programs to be executed by a computer system with memory and one or more processors, the one or more programs comprising:

instructions for receiving from a first client device a user request to clarify audible verbal information of a media content item playing on a second client device in proximity to the first client device, wherein the user request includes an audio sample of the media content item and a user query issued to the first client device by a user, the user query regarding a subset, not all, of audio content of the media content item; and

in response to the user request;

instructions for identifying the media content item based on the audio sample;

instructions for identifying a portion of the media content containing the subset of audio content based on the audio sample and content of the user query;

instructions for, in accordance with the identity of the media content item, obtaining textual information responsive to the user query for the portion of the media content item; and

instructions for transmitting to the first client device at least a portion of the textual information for presentation at the first client device.

19. The computer readable storage medium of claim 18, wherein the audible verbal information is associated with a plurality of speakers;

and the user request comprises a request to clarify the audible verbal information with respect to a respective speaker.

20. The computer readable storage medium of claim 18, further comprising:

instructions for receiving from the first client device a second user request to clarify audible verbal information associated with the media content item playing on the second media device in proximity to the first client device, wherein the second user request includes a second audio sample of the media content item and a second user query issued to the first client device by the user, wherein the second audio sample corresponds to a second portion of the audio content of the media content item output during the playback of the media content item by the second client device proximate in time to the issuance of the second user query and recorded by the first client device; and

in response to the second user request;

instructions for identifying a second playback position in the media content corresponding to the second audio sample;

instructions for, in accordance with the second playback position and the identity of the media content item, obtaining second textual information corresponding to the second user query for a second portion of the media content item; and

instructions for transmitting to the first client device at least a portion of the second textual information for presentation at the first client device.

21. The computer readable storage medium of claim 18, further comprising:

instructions for obtaining or generating a translation of the textual information; and

instructions for transmitting to the client device at least a portion of the translation, wherein the at least a portion of the translation corresponds to the at least a portion of the textual information.