Output of content based on speech-based searching and browsing requests

• US 10,056,078 B1
• Filed: 06/28/2016
• Issued: 08/21/2018
• Est. Priority Date: 06/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Shepherd, Benjamin Lee, Fernandez, Joe Virgil, Gangadaran, Vijayabaskar, McManamon, Mallory Ann, Bentley, Peter Van Tuyl, Srivastava, Shailesh
• Primary Examiner(s): Nguyen Ba, Hoang Vu A
• Application Number:
  US15/195,048
• Time in Patent Office: 784 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/632 Query formulation
• CPC Class Codes: G06F 16/732 Query formulation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06Q 20/12 specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/123 Shopping for digital content
• CPC Class Codes: G06Q 20/306 using TV related infrastructures external card used for payment with client device H04N21/4185
• CPC Class Codes: G06Q 20/3272 using an audio code
• CPC Class Codes: G06Q 20/36 using electronic wallets or electronic money safes
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4826 using recommendation lists, e.g. of programs or channels sorted out according to their score
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739

Abstract

Speech-controlled searching and browsing for content using speech-controlled devices, or other input-limited devices, is described. A user may audibly indicate to a speech-controlled device whether the user wants to search or browse for content, along with a topic of the content/results to be retrieved. A server, located remotely from the speech-controlled device determines an appropriate endpoint device for displaying results of the requested search or browse. The server also determines an appropriate content source for the requested content, and sends a request for the content to the content source. The server receives search or browse results from the content source and forwards them to the determined endpoint device.

Claims

1. A computer implemented method comprising:

receiving, from a speech-controlled device, first input audio data;

performing speech processing on the first input audio data to create first text data;

determining the first text data corresponds to a first request to browse first content corresponding to a topic;

determining that profile data associated with the speech-controlled device is further associated with a plurality of output devices including a first display device;

selecting, from among the plurality of output devices, the first display device for outputting first results specific to the topic based on the first display device being associated with a first template for outputting results;

sending, to a content source, a first data request representing the topic;

receiving first results data from the content source, the first results data representing first content authorized for playback by the first display device;

generating first metadata including a command representing how the first results data is to be presented based on the first template;

sending, to the first display device, the first results data; and

sending, to the first display device, the first metadata.

2. The computer-implemented method of claim 1, further comprising:

receiving context information from the speech-controlled device, the context information being gathered when first input audio corresponding to the first input audio data was received by the speech-controlled device, the context information identifying at least one of a type of user, location of the speech-controlled device, or time of day; and

including the context information within the first data request,

wherein the first results data is further received based on the context information.

3. The computer-implemented method of claim 1, further comprising:

receiving, from the speech-controlled device, second input audio data;

performing speech processing on the second input audio data to create second text data;

determining the second text data corresponds to a second request to refine the first results data presented by the first display device;

determining the second text data includes second content identifying information specific to a portion of the first results data;

generating second results data using the second content identifying information;

generating second metadata including a second command representing how the second results data is to be presented based on the first template;

sending, to the first display device, the second results data;

sending, to the first display device, the second metadata.

4. The computer-implemented method of claim 1, further comprising:

receiving, from the speech-controlled device, second input audio data;

performing speech processing on the second input audio data to create second text data;

determining the second text data corresponds to a second request to supplement the first results data with additional results;

determining the second text data includes second content identifying information;

sending, to the content source, a second data request representing the second content identifying information;

receiving, from the content source, second results data specific to the second content identifying information;

generating second metadata including a second command representing how the second results data is to be presented;

sending, to the first display device, the second results data; and

sending, to the first display device, the second metadata.

5. A system comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the system to;

receive, from a first device, first audio data corresponding to a first spoken utterance;

perform speech processing on the first audio data to generate first text data;

determine the first text data corresponds to a browse request including a browse topic encompassing multiple related content;

determine a plurality of output devices associated with the first device;

determine, from among the plurality of output devices, a second device is associated with a first output template, the first output template indicating how browse results are to be presented via a display of the second device;

based at least in part on determining the second device is associated with the first output template, determine that the second device is to output results of the browse request;

send, to a content source, a first data request that includes information specific to the browse topic and the first output template;

receive, from the content source, first results data specific to the browse topic and including first visual data specific to the first output template;

cause the second device to present at least a portion of the first results data;

receive, from the first device, second audio data corresponding to a second spoken utterance;

perform speech processing on the second audio data to generate second text data;

determine the second text data corresponds to a search request including a search topic specific to a particular item;

determine the second device is associated with a second output template, the second output template indicating how search results are to be presented via the display, the second output template and the first output template indicating different information to be presented via the display;

based at least in part on determining the second device is associated with the second output template, determine that the second device is to output results of the search request;

send, to the content source, a second data request that includes information specific to the search topic and the second output template;

receive, from the content source, second results data specific to the search topic and including second visual data specific to the second output template; and

cause the second device to present at least a portion of the second results data.

6. The system of claim 5, wherein the instructions, when executed by the at least one processor, causing the system to determine the second text data corresponds to the search request further include instructions to:

determine a field of a data structure of the content source, the field being used for querying the content source; and

associate a first portion of the second text data with the field.

7. The system of claim 5, wherein the instructions, when executed by the at least one processor, further cause the system to:

generate metadata including a command representing how the at least a portion of the first results data is to be presented based on the first output template; and

send the first results data to the second device; and

send the metadata to the second device.

8. The system of claim 5, wherein the instructions, when executed by the at least one processor, further cause the system to:

receive, from the first device, third audio data;

determine the third audio data corresponds to a request to refine presentment of the at least a portion of the first results data; and

cause the second device to present a refined portion of the at least a portion of the first results data.

9. The system of claim 5, wherein the instructions, when executed by the at least one processor, further cause the system to:

receive, from the first device, third audio data;

determine the third audio data corresponds to a request to present additional results associated with the browse topic; and

cause the second device to present at least a second portion of the first results data missing from the at least a portion of the first results data.

10. The system of claim 9, wherein the instructions, when executed by the at least one processor, further cause the system to:

determine the additional results were missing from the first results data received from the content source;

send, to the content source, a third data request that includes information specific to the additional results;

receive, from the content source, additional results data; and

cause the second device to present the additional results data.

11. The system of claim 9, wherein the instructions, when executed by the at least one processor, further cause the system to:

determine the additional results were included in the first results data received from the content source;

determine the additional results were missing from the at least a portion of the first results data; and

cause the second device to present at least a second portion of the first results data corresponding to the additional results.

12. The system of claim 5, wherein the instructions, when executed by the at least one processor, further cause the system to:

determine user profile data, associated with the first device, represents a plurality of content sources including the content source; and

determined the user profile data represents the plurality of output devices.

13. A computer-implemented method comprising:

receiving, from a first device, first audio data corresponding to a first spoken utterance;

performing speech processing on the first audio data to generate first text data;

determining the first text data corresponds to a browse request including a browse topic encompassing multiple related content;

determining a plurality of output devices associated with the first device;

determining, from among the plurality of output devices, a second device is associated with a first output template, the first output template indicating how browse results are to be presented via a display of the second device;

based at least in part on determining the second device is associated with the first output template, determining that the second device is to output results of the browse request;

sending, to a content source, a first data request that includes information specific to the browse topic and the first output template;

receiving, from the content source, first results data specific to the browse topic and including first visual data specific to the first output template;

causing the second device to present at least a portion of the first results data;

receiving, from the first device, second audio data corresponding to a second spoken utterance;

performing speech processing on the second audio data to generate second text data;

determining the second text data corresponds to a search request including a search topic specific to a particular item;

determining the second device is associated with a second output template, the second output template indicating how search results are to be presented via the display, the second output template and the first output template indicating different information to be presented via the display;

based at least in part on determining the second device is associated with the second output template, determining that the second device is to output results of the search request;

sending, to the content source, a second data request that includes information specific to the search topic and the search output template;

receiving, from the content source, second results data specific to the search topic and including second visual data specific to the second output template; and

causing the second device to present at least a portion of the second results data.

14. The computer-implemented method of claim 13, wherein determining the second text data corresponds to the search request comprises:

determining a field of a data structure of the content source, the field being used for querying the content source; and

associating a first portion of the second text data with the field.

15. The computer-implemented method of claim 13, further comprising:

generating metadata including a command representing how the at least a portion of the first results data is to be presented based on the first output template;

sending the first results data to the second device; and

sending the metadata to the second device.

16. The computer-implemented method of claim 13, further comprising:

receiving, from the first device, third audio data;

determining the third audio data corresponds to a request to refine presentment of the at least a portion of the first results data; and

causing the second device to present a refined portion of the at least a portion of the first results data.

17. The computer-implemented method of claim 13, further comprising:

receiving, from the first device, third audio data;

determining the third audio data corresponds to a request to present additional results associated with the browse topic; and

causing the second device to present at least a second portion of the first results data missing from the at least a portion of the first results data.

18. The computer-implemented method of claim 17, further comprising:

determining the additional results were missing from the first results data received from the content source;

sending, to the content source, a third data request that includes information specific to the additional results;

receiving, from the content source, additional results data; and

causing the second device to present the additional results data.

19. The computer-implemented method of claim 17, further comprising:

determining the additional results were included in the first results data received from the content source;

determining the additional results were missing from the at least a portion of the first results data; and

causing the second device to present at least a second portion of the first results data corresponding to the additional results.

20. The computer-implemented method of claim 13, further comprising:

determining user profile data, associated with the first device, represents a plurality of content sources including the content source; and

determining, in the user profile data represents the plurality of output devices.