Data ingestion pipeline

• US 10,108,707 B1
• Filed: 09/22/2017
• Issued: 10/23/2018
• Est. Priority Date: 09/22/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Chu, Yahui, Whitney, Stephen Allen
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US15/712,571
• Time in Patent Office: 396 Days
• Field of Search: 700 84, 704705, 704275, 704270, 707705, 707722, 707732, 455563, 709244, 709260
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/635 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Techniques for expanding system capabilities to execute user commands relating to trending topics (e.g., real-time news questions, trending questions, sports questions, game questions, politic questions, etc.) are described. The system gathers data from a variety of sources (e.g., news feeds, social media feeds, RSS feeds, news websites, etc.). The system segments gathered data corresponding to, for example, topic and or entity. The system may only store data corresponding to a topic or entity in a dedicated trending storage if the system receives data corresponding to the topic or entity from a number of different sources satisfying a threshold number of sources. Data in the dedicated trending storage may be maintained using decay models or algorithms. For example, the more often the system receives data corresponding to a topic or entity from one or more sources, the longer the data is maintained in the storage, and vice versa.

Claims

1. A computer-implemented method comprising:

during a first period of time;

receiving, from a first remote device, first content data associated with a topic;

receiving, from a second remote device, second content data associated with the topic;

based on the first content data and the second content data both being associated with the topic, grouping the first content data and the second content data to generate first grouped data;

determining the first remote device and the second remote device correspond to a number of remote devices satisfying a threshold number of remote devices;

storing, based on the number satisfying the threshold number, the first grouped data as first stored data;

during a second period of time after the first period of time;

receiving, from a first device, input audio data corresponding to an utterance;

performing speech processing on the input audio data to determine a command corresponding to the topic;

determining, in a profile associated with the first device, a preferred content source associated with the topic;

determining the first remote device corresponds to the preferred content source;

performing text-to-speech (TTS) processing on the first content data to generate output audio data; and

causing the first device to emit audio corresponding to the output audio data.

2. The computer-implemented method of claim 1, further comprising:

determining a number of sources that, within a time period, publish data corresponding to the topic;

determining that the number falls below a frequency threshold; and

determining the first stored data is no longer trending.

3. The computer-implemented method of claim 1, further comprising:

determining, during a first time period, a first number of sources from which a first plurality of data corresponding to the topic is received;

determining, during a second time period, a second number of sources from which a second plurality of data corresponding to the topic is received;

determining the second number is less than the first number; and

determining, based on the second number being less than the first number, a future time when the first stored data is to be deleted from a trending knowledge base.

4. The computer-implemented method of claim 3, further comprising:

storing the first grouped data in a general knowledge base; and

permitting the first grouped data to persist in the general knowledge base after the future time.

5. A system comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the system to;

receive input data;

perform speech processing on the input data to determine the input data corresponds to a topic;

determine, from profile data associated with a device, a preferred content source associated with the topic;

determine stored data corresponding to the topic, the stored data being received from a number of content sources satisfying a threshold number of content sources;

determine at least a portion of the stored data received from the preferred content source; and

cause the device to output content corresponding to the at least a portion.

6. The system of claim 5, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

receive first text data from a first content source;

receive second text data from a second content source;

perform natural language processing on the first text data to determine the topic;

perform natural language processing on the second text data to determine the topic;

determine the number to include at least to the first content source and the second content source;

determine the number satisfies the threshold number; and

generate, based on the number satisfying the threshold number, the stored data to include the first text data and the second text data.

7. The system of claim 5, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

determine a number of content sources that, within a time period, publish data corresponding to the topic;

determine that the number falls below a frequency threshold; and

based on the number falling below the frequency threshold, determine a future time that the stored data is to be deleted from a trending knowledge.

8. The system of claim 7, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

store the stored data in a general knowledge base as second stored data; and

permit the second stored data to persist in the general knowledge base after the future time.

9. The system of claim 5, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

determine, during a first time period, a first number of content sources from which a first plurality of data corresponding to the topic is received;

determine, during a second time period, a second number of content sources from which a second plurality of data corresponding to the topic is received;

determine the second number is less than the first number; and

determine, based on the second number being less than the first number, a future time when the stored data is to be deleted.

10. The system of claim 5, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

perform natural language processing on the stored data to generate natural language results;

generate, based on the natural language results, a synopsis of the stored data; and

cause the device to output content corresponding to the synopsis.

11. The system of claim 5, wherein the stored data corresponds to at least one of website content data, news data, news feed data, podcast data, or rich site summary (RSS) feed data.

12. The system of claim 5, wherein the stored data includes a plurality of data files received from a single content source, the plurality of data files satisfying a threshold number of data files.

13. A computer-implemented method comprising:

receiving input data;

performing speech processing on the input data to determine the input data corresponds to a topic;

determining, from profile data associated with a device, a preferred content source associated with the topic;

determining stored data corresponding to the topic, the stored data being received from a number of content sources satisfying a threshold number of content sources;

determining at least a portion of the stored data received from the preferred content source; and

causing the device to output content corresponding to the at least a portion.

14. The computer-implemented method of claim 13, further comprising

receiving first text data from a first content source;

receiving second text data from a second content source;

performing natural language processing on the first text data to determine the topic;

performing natural language processing on the second text data to determine the topic;

determining the number to include at least to the first content source and the second content source;

determining the number satisfies the threshold number; and

generating, based on the number satisfying the threshold number, the stored data to include the first text data and the second text data.

15. The computer-implemented method of claim 13, further comprising:

determining a number of content sources that, within a time period, publish data corresponding to the topic;

determining that the number falls below a frequency threshold; and

based on the number falling below the frequency threshold, determine a future time that the stored data is to be deleted from a trending knowledge base.

16. The computer-implemented method of claim 15, further comprising:

storing the stored data in a general knowledge base as second stored data; and

permitting the second stored data to persist in the general knowledge base after the future time.

17. The computer-implemented method of claim 13, further comprising:

determining, during a first time period, a first number of content sources from which a first plurality of data corresponding to the topic is received;

determining, determine a second time period, a second number of content sources from which a second plurality of data corresponding to the topic is received;

determining the second number is less than the first number; and

determining, based on the second number being less than the first number, a future time when the stored data is to be deleted.

18. The computer-implemented method of claim 13, further comprising:

performing natural language processing on the stored data to generate natural language results;

generating, based on the natural language results, a synopsis of the stored data; and

causing the device to output content corresponding to the synopsis.

19. The computer-implemented method of claim 13, wherein the stored data corresponds to at least one of web site content data, news data, news feed data, podcast data, or rich site summary (RSS) feed data.

20. The computer-implemented method of claim 13, wherein the stored data includes a plurality of data files received from a single content source, the plurality of data files satisfying a threshold number of data files.