Answering questions using environmental context

• US 9,576,576 B2
• Filed: 08/01/2016
• Issued: 02/21/2017
• Est. Priority Date: 09/10/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Postelnicu, Gheorghe
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US15/224,944
• Publication Number:
  US 20160343371A1
• Time in Patent Office: 204 Days
• Field of Search: 704/233, 704/235, 704/243, 704/226, 704/205, 704/246, 704/247, 704/251, 704/252, 704/270, 704/270.1, 704/275
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/686 using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving audio data encoding an utterance and environmental data, obtaining a transcription of the utterance, identifying an entity using the environmental data, submitting a query to a natural language query processing engine, wherein the query includes at least a portion of the transcription and data that identifies the entity, and obtaining one or more results of the query.

Claims

1. A computer-implemented method comprising:

generating, by a mobile device, an audio recording of (i) a spoken, natural language question that has been asked about an item of media content and that does not name, or request the name of, the item of media content, and (ii) ambient sounds that are associated with the playback of the item of media content and that are recorded contemporaneously with the question being asked;

forwarding the audio recording to a coordination engine server;

receiving from the coordination engine server an answer to the question, the answer based on processing of the question by a query processing engine server and on processing of the ambient sounds by a content identification engine server; and

in response to the question, providing, by the mobile device, an answer to the question that has been asked about the item of media content.

2. The computer-implemented method of claim 1, comprising:

identifying one or more keywords corresponding to the question,

associating the one or more keywords with one or more types of media content, and

providing the answer based on the question and the one or more types of media content.

3. The computer-implemented method of claim 2, wherein the one or more types of media content includes at least one of movie, music, television show, audio podcast, image, artwork, book, magazine, trailer, video, podcast, Internet video and video game.

4. The computer-implemented method of claim 2, wherein providing the answer based on the one or more types of media content further comprises:

identifying two or more candidate answers of the question,

generating ranked scores for each of the two or more candidate answers, the ranked scores based on the one or more types of media content, and

providing the answer based on the question and the ranked scores.

5. The computer-implemented method of claim 1, further comprising streaming the ambient sounds.

6. The computer-implemented method of claim 1, wherein the coordination engine server, the query processing engine server and the content identification server are all the same server.

7. The computer-implemented method of claim 1, further comprising:

detecting environmental image data associated with the item of media content, and

providing the answer based on the question and the environmental image data.

8. The computer-implemented method of claim 7, further comprising:

identifying one or more types of media content based on the environmental image data, and

providing the answer based on the question, the environmental image data and the one or more types of media content.

9. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

generating, by a mobile device, an audio recording of (i) a spoken, natural language question that has been asked about an item of media content and that does not name, or request the name of, the item of media content, and (ii) ambient sounds that are associated with the playback of the item of media content and that are recorded contemporaneously with the question being asked;

forwarding the audio recording to a coordination engine server;

receiving from the coordination engine server an answer to the question, the answer based on processing of the question by a query processing engine server and on processing of the ambient sounds by a content identification engine server; and

in response to the question, providing, by the mobile device, an answer to the question that has been asked about the item of media content.

10. The system of claim 9, comprising:

identifying one or more keywords corresponding to the question,

associating the one or more keywords with one or more types of media content, and

providing the answer based on the question and the one or more types of media content.

11. The system of claim 10, wherein the one or more types of media content includes at least one of movie, music, television show, audio podcast, image, artwork, book, magazine, trailer, video, podcast, Internet video and video game.

12. The system of claim 10, wherein providing the answer based on the one or more types of media content further comprises:

identifying two or more candidate answers of the question,

generating ranked scores for each of the two or more candidate answers, the ranked scores based on the one or more types of media content, and

providing the answer based on the question and the ranked scores.

13. The system of claim 10, further comprising streaming the ambient sounds.

14. The system of claim 10, wherein the coordination engine server, the query processing engine server and the content identification server are all the same server.

15. The system of claim 10, further comprising:

detecting environmental image data associated with the item of media content, and

providing the answer based on the question and the environmental image data.

16. The system of claim 15, further comprising:

identifying one or more types of media content based on the environmental image data, and

providing the answer based on the question, the environmental image data and the one or more types of media content.

17. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

generating, an audio recording of (i) a spoken, natural language question that has been asked about an item of media content and that does not name, or request the name of, the item of media content, and (ii) ambient sounds that are associated with the playback of the item of media content and that are recorded contemporaneously with the question being asked;

forwarding the audio recording to a coordination engine server;

receiving from the coordination engine server an answer to the question, the answer based on processing of the question by a query processing engine server and on processing of the ambient sounds by a content identification engine server; and

in response to the question, providing an answer to the question that has been asked about the item of media content.

18. The non-transitory computer-readable medium of claim 17, comprising:

identifying one or more keywords corresponding to the question,

associating the one or more keywords with one or more types of media content, and

providing the answer based on the question and the one or more types of media content.

19. The non-transitory computer-readable medium of claim 17, further comprising streaming the ambient sounds.

20. The non-transitory computer-readable medium of claim 17, further comprising:

detecting environmental image data associated with the item of media content, and

providing the answer based on the question and the environmental image data.

21. The non-transitory computer-readable medium of claim 17, wherein the coordination engine server, the query processing engine server and the content identification server are all the same server.

22. A computer-implemented method comprising:

generating, by a mobile device, an audio recording of (i) a spoken, natural language question that has been asked about an item of media content and that does not name, or request the name of, the item of media content, and (ii) ambient sounds that are associated with the playback of the item of media content and that are recorded contemporaneously with the question being asked;

forwarding the audio recording to a coordination engine server;

receiving from the coordination engine server an answer to the question, the answer based on processing of a transcription of the question by a query processing engine server and on processing of the ambient sounds by a content identification engine server; and

in response to the question, providing, by the mobile device, an answer to the question that has been asked about the item of media content.