Speech recognition for localized content

• US 10,448,115 B1
• Filed: 09/28/2016
• Issued: 10/15/2019
• Est. Priority Date: 09/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Jamal, Shah Pavel, Lawrence, Mark, Halbach, Kyle Daniel, Peacock, Neal, Quach, Anderson Kailodge
• Primary Examiner(s): Peng, Hsiungfei
• Application Number:
  US15/278,374
• Time in Patent Office: 1,112 Days
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/4383 Accessing a communication channel
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/462 Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
• CPC Class Codes: H04N 21/4825 using a list of items to be played back in a given order, e.g. playlists
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/6112 involving terrestrial transmission, e.g. DVB-T
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

A system capable of generating and storing program data associated with localized content, such as over-the-air television broadcasts. The system may update a knowledge base with the program data to improve natural language understanding (NLU) processing associated with the localized content. For example, a voice enabled device may receive a voice command and the system may perform NLU processing to interpret the voice command, determine content based on the program data and send an instruction to an output device to output the content. The program data may include information about localized content such as channels detected by the output device and/or content broadcast on the channels, including information about an output device, a channel number, call sign, station name, signal strength, etc. associated with a detected channel, and/or a program name, genre, actors, directors, producers, etc. associated with content broadcast on the channel or stored in a local storage device.

Claims

1. A computer-implemented method, the method comprising:

receiving channel data that represents a plurality of terrestrial over-the-air broadcast television channels detectable by a first device associated with a user profile, the plurality of terrestrial over-the-air broadcast television channels including a first broadcast channel;

determining, using the channel data and a program guide data source, program data including;

a list of program titles that are broadcast on the plurality of terrestrial over-the-air broadcast television channels, and

a list of broadcast time periods corresponding to each of the program titles;

receiving first audio input data from a second device associated with the user profile;

performing speech recognition on the first audio input data to determine text data;

determining, using the user profile, that the second device is associated with the first device;

configuring, using information included in the program data, a knowledge database associated with the user profile, wherein the knowledge database is used by a natural language understanding (NLU) component to interpret the text data;

determining, using the text data, the NLU component, and the knowledge database, that the text data corresponds to a command to operate the first device to output a program received using the first broadcast channel; and

sending, to the first device, an instruction to tune to the first broadcast channel and output the program.

2. The computer-implemented method of claim 1, further comprising:

determining a channel identification associated with the first broadcast channel;

determining a call sign associated with the first broadcast channel;

determining a first program title of the program broadcast on the first broadcast channel;

determining a list of actors associated with the program;

determining a genre associated with the program; and

configuring the knowledge database to include the channel identification, the call sign, the first program title, each actor included in the list of actors, and the genre.

3. The computer-implemented method of claim 1, wherein:

the configuring the knowledge database further comprises assigning ranking values to words included in the program data;

the determining that the text corresponds to the command further comprises;

identifying, using the NLU component a text string included in the text data,

determining, using the NLU component, that the text string is an entity,

determining, using the NLU component, that the text string is included in the knowledge database,

determining, using the NLU component and the knowledge database, a first ranking value corresponding to the text string,

determining that the first ranking value exceeds a threshold, and

determining that the text string corresponds to a title of the program.

4. The computer-implemented method of claim 1, wherein the determining that the text data corresponds to the command further comprises:

determining that the text data includes a first word;

identifying the first word within the program data;

determining that the first word corresponds to a channel identification associated with the first broadcast channel;

determining a confidence score that the text data is associated with the program, wherein the confidence score is determined based at least in part on the first word corresponding to the channel identification; and

determining that the confidence score is above a threshold.

5. A computer-implemented method, comprising:

receiving first data indicating a plurality of content channels accessible via a first device associated with a user profile, the plurality of content channels accessible by detecting terrestrial over-the-air broadcasts and including a content channel;

determining, based on the first data, second data including information associated with the plurality of content channels;

associating the second data with the user profile;

operating a speech processing component based at least in part on the second data, wherein the speech processing component is associated with the user profile;

receiving audio data representing an utterance;

performing speech recognition on the audio data to determine text data;

determining that the audio data is associated with the first device;

performing natural language understanding, using the text data and the speech processing component, to determine that the text data corresponds to a command to operate the first device to output a first content item being broadcast on the content channel; and

sending a message to the first device to tune to the content channel.

6. The computer-implemented method of claim 5, further comprising, based on the second data:

determining a channel identification associated with the content channel;

determining a call sign associated with the content channel;

determining a title of the first content item;

determining a list of actors associated with the first content item; and

determining a genre associated with the first content item; and

updating the user profile to include the channel identification, the call sign, the title, each actor included in the list of actors, and the genre.

7. The computer-implemented method of claim 5, further comprising:

configuring an entity resolution component to recognize the information included in the second data; and

performing the natural language understanding, using the text data, the speech processing component, and the entity resolution component, to determine that the text data corresponds to the command to operate the first device to output the first content item.

8. The computer-implemented method of claim 5, further comprising:

updating the user profile to assign ranking values to words included in the second data;

identifying a text string included in the text data;

determining that the text string is an entity;

determining that the text string is included in the user profile;

determining a first ranking value corresponding to the text string;

determining that the first ranking value exceeds a threshold; and

determining that the text string corresponds to a title of the first content item.

9. The computer-implemented method of claim 5, further comprising:

receiving the audio data from the first device during a first time period;

receiving the first data from the first device during the first time period; and

sending, to the first device, the message to tune to the content channel.

10. The computer-implemented method of claim 5, further comprising:

receiving the first data from the first device;

receiving the audio data from a second device;

determining that the second device is associated with the user profile;

determining, based at least in part on the user profile, that the audio data is associated with the first device; and

sending, to the first device, the message to tune to the content channel.

11. The computer-implemented method of claim 5, wherein the performing natural language understanding further comprises:

determining, using the text data and the speech processing component, a first confidence score that the text data corresponds to the command;

determining, using the text data and the speech processing component, a second confidence score that the text data corresponds to a second command to operate the first device to output a second content item;

determining that the first confidence score is above a threshold;

determining that the second confidence score is above the threshold;

sending, to the first device, a second message to display a user interface listing the first content item and the second content item; and

receiving, from the first device, an indication of the first content item.

12. The computer-implemented method of claim 5, wherein the determining that the text data corresponds to the command further comprises:

determining that the text data includes a first word;

identifying the first word within the user profile;

determining that the first word is associated with the first content item;

determining a confidence score that the text data is associated with the first content item, wherein the confidence score is determined based at least in part on the first word being associated with the first content item; and

determining that the confidence score is above a threshold.

13. The computer-implemented method of claim 5, wherein:

the first data indicates a channel identification associated with the content channel; and

the second data includes at least one of;

the channel identification associated with the content channel,

a call sign associated with the content channel,

a title of the first content item,

a list of actors associated with the first content item, and

a broadcast time period associated with the first content item.

14. The computer-implemented method of claim 5, wherein the message includes a command instructing the first device to control an over-the-air television tuner to tune to the content channel.

15. The computer-implemented method of claim 5, further comprising:

configuring the speech processing component based at least in part on the second data, wherein the speech processing component is associated with the user profile.

16. The computer-implemented method of claim 5, wherein the plurality of content channels are accessible by detecting the terrestrial over-the-air broadcasts based on a signal strength.

17. A system, comprising:

at least one processor;

memory including instructions operable to be executed by the at least one processor to perform a set of actions to cause the system to;

receive first data indicating a plurality of content channels accessible via a first device associated with a user profile, the plurality of content channels accessible by detecting over-the-air broadcasts and including a content channel;

determine, based on the first data, second data including information associated with the plurality of content channels;

associate the second data with the user profile;

receive audio data representing an utterance;

perform speech recognition on the audio data to determine text data;

determine that the audio data is associated with the first device;

perform natural language understanding, using the text data and the user profile, to determine that the text data corresponds to a command to operate the first device to output a first content item being broadcast on the content channel, wherein determining that the text data corresponds to the command further comprises;

determining that the text data includes a first word,

identifying the first word within the user profile,

determining that the first word is associated with the first content item,

determining a confidence score that the text data is associated with the first content item, wherein the confidence score is determined based at least in part on the first word being associated with the first content item, and

determining that the confidence score is above a threshold; and

send a message to the first device to tune to the content channel.

18. The system of claim 17, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to, based on the second data:

determine a channel identification associated with the content channel;

determine a call sign associated with the content channel;

determine a title of the first content item;

determine a list of actors associated with the first content item;

determine a genre associated with the first content item; and

update the user profile to include the channel identification, the call sign, the title, each actor included in the list of actors, and the genre.

19. The system of claim 17, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

configure an entity resolution component to recognize the information included in the second data; and

perform the natural language understanding, using the text data, the user profile, and the entity resolution component, to determine that the text data corresponds to the command to operate the first device to output the first content item.

20. The system of claim 17, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

update the user profile to assign ranking values to words included in the second data;

identify a text string included in the text data;

determine that the text string is an entity;

determine that the text string is included in the user profile;

determine a first ranking value corresponding to the text string;

determine that the first ranking value exceeds a second threshold; and

determine that the text string corresponds to a title of the first content item.

21. The system of claim 17, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

receive the audio data from the first device during a first time period;

receive the first data from the first device during the first time period; and

send, to the first device, the message to tune to the content channel.

22. The system of claim 17, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

receive the first data from the first device;

receive the audio data from a second device;

determine that the second device is associated with the user profile;

determine, based at least in part on the user profile, that the audio data is associated with the first device; and

send, to the first device, the message to tune to the content channel.

23. The system of claim 17, wherein the memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine, using the text data and the user profile, a second confidence score that the text data corresponds to the command;

determine, using the text data and the user profile, a third confidence score that the text data corresponds to a second command to operate the first device to output a second content item;

determine that the second confidence score is above a second threshold;

determine that the third confidence score is above the second threshold;

send, to the first device, a second message to display a user interface listing the first content item and the second content item; and

receive, from the first device, an indication of the first content item.

24. A computer-implemented method, comprising:

receiving, from a first device associated with a user profile, first data indicating a plurality of content channels accessible via the first device by detecting over-the-air broadcasts, the plurality of content channels including a content channel;

receiving, from a second device, second data including information associated with a second plurality of content channels, the second plurality of content channels including the plurality of content channels;

determining, based on the first data, third data including information associated with the plurality of content channels, wherein the third data corresponds to a portion of the second data that is associated with the plurality of content channels;

associating the third data with the user profile;

operating a speech processing component based at least in part on the third data, wherein the speech processing component is associated with the user profile;

receiving audio data representing an utterance;

performing speech recognition on the audio data to determine text data;

determining that the audio data is associated with the first device;

performing natural language understanding, using the text data and the speech processing component, to determine that the text data corresponds to a command to operate the first device to output a first content item being broadcast on the content channel; and

sending a message to the first device to tune to the content channel.