Data stream ingestion and persistence techniques

• US 9,858,322 B2
• Filed: 11/11/2013
• Issued: 01/02/2018
• Est. Priority Date: 11/11/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Theimer, Marvin Michael, Ghare, Gaurav D., Dunagan, John David, Burgess, Greg, Xiong, Ying
• Primary Examiner(s): Roumi, Mahran Abu
• Assistant Examiner(s): Ye, Zi
• Application Number:
  US14/077,162
• Publication Number:
  US 20150134795A1
• Time in Patent Office: 1,513 Days
• Field of Search: 709223, 707661
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A programmatic interface is implemented, enabling a client of a stream management service to select a data ingestion policy for a data stream. A client request selecting an at-least-once ingestion policy is received. In accordance with the at-least-once policy, a client may transmit an indication of a data record one or more times to the service until a positive acknowledgement is received. In response to receiving a plurality of transmissions indicating a particular data record, respective positive acknowledgements are sent to the client. Based on a persistence policy selected for the stream, copies of the data record are stored at one or more storage locations in response to one particular transmission of the plurality of transmissions.

Claims

1. A system, comprising:

one or more computing devices comprising one or more processors and memory and configured to;

implement a first set of programmatic interfaces enabling a client of a multi-tenant stream management service to select, for a particular data stream, a data ingestion policy from among a plurality of data ingestion policies, wherein the plurality of data ingestion policies includes an at-least-once ingestion policy in accordance with which a record submitter transmits an indication of a data record one or more times to the stream management service until a positive acknowledgement is received;

implement a second set of programmatic interfaces enabling the client to select, for the particular data stream, a data persistence policy from among a plurality of data persistence policies, wherein the plurality of data persistence policies comprises a multiple-replica persistence policy, in accordance with which multiple copies of the data record are to be stored at respective storage locations by the stream management service;

receive, at the stream management service via respective programmatic interfaces of the first and second set, a first indication that the client has selected the at-least-once ingestion policy for the particular data stream and a second indication that the client has selected the multiple-replica persistence policy for the particular data stream;

determine a number of data ingestion nodes or a number of data storage nodes to be configured for the particular data stream based at least in part on a partitioning policy in accordance with which a data ingestion node of the number of data ingestion nodes is selected to ingest data records of a particular partition of the particular data stream or a data storage node of the number of data storage nodes is selected to store data records of the particular partition of the particular data stream; and

in response to a plurality of transmissions indicating a particular data record to the stream management service,

send at least one positive acknowledgement corresponding to the plurality of transmissions in accordance with the at-least-once ingestion policy; and

store, in response to a particular transmission of the plurality of transmissions, copies of the particular data record at a plurality of storage locations in accordance with the multiple-replica persistence policy.

2. The system as recited in claim 1, wherein the plurality of data ingestion policies comprises a best-effort ingestion policy, in accordance with which the stream management service is to accept and store at least some data records of a specified stream without providing corresponding positive acknowledgements to a record submitter.

3. The system as recited in claim 1, wherein the multiple-replica persistence policy selected by the client comprises an indication of a type of storage location to be used for storing copies of data records, wherein the type of storage location comprises one of:

(a) magnetic disk-based storage, (b) solid state drive (SSD), (c) volatile RAM (random access memory), (d) non-volatile RAM, (e) a database management system, or (f) a storage node of a network-accessible storage service implemented by a provider network.

4. The system as recited in claim 1, wherein the multiple-replica persistence policy selected by the client comprises an indication of a requested data durability level, wherein the one or more computing devices are further configured to:

select the plurality of storage locations based at least in part on the requested data durability level.

5. The system as recited in claim 1, wherein the one or more computing devices are further configured to:

determine a billing amount that a particular client is to be charged for stream management operations based at least in part on one or more of;

(a) a data ingestion policy selected by the particular client, and (b) a data persistence policy selected by the particular client.

6. A method, comprising:

performing, by one or more computing devices;

implementing a set of programmatic interfaces enabling a client of a stream management service to select, for a particular data stream, a data ingestion policy from among a plurality of data ingestion policies, wherein the plurality of data ingestion policies includes an at-least-once ingestion policy in accordance with which a record submitter is to transmit an indication of a data record one or more times to the stream management service until a positive acknowledgement is received;

receiving a request via a programmatic interface of the set, indicating that the client has selected the at-least-once ingestion policy for the particular data stream;

determining a number of data ingestion nodes to be configured for the particular data stream based at least in part on a partitioning policy in accordance with which a data ingestion node of the number of data ingestion nodes is selected to ingest data records of a particular partition of the particular data stream; and

in response to receiving a plurality of transmissions indicating a particular data record at the stream management service,

sending respective positive acknowledgements corresponding to each transmission of the plurality of transmissions in accordance with the at-least-once ingestion policy; and

storing, in response to receiving a particular transmission of the plurality of transmissions, copies of the particular data record at a one or more storage locations in accordance with a data persistence policy selected for the particular data stream.

7. The method as recited in claim 6, wherein the plurality of data ingestion policies comprises a best-effort ingestion policy, in accordance with which the stream management service is to accept and store at least some data records of a specified stream without providing corresponding positive acknowledgements to a record submitter.

8. The method as recited in claim 6, wherein in accordance with the at-least-once ingestion policy, the stream management service is configured to eliminate duplicated data records.

9. The method as recited in claim 6, wherein in accordance with the at-least-once ingestion policy, the stream management service is configured to store, corresponding to the plurality of transmissions, more than one copy of the particular data record.

10. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

implementing a second set of programmatic interfaces enabling the client to select, for the particular data stream, the data persistence policy from among a plurality of data persistence policies.

11. The method as recited in claim 10, wherein the plurality of data persistence policies include a multiple-replica persistence policy and a single-replica persistence policy.

12. The method as recited in claim 10, wherein the data persistence policy comprises an indication of a type of storage location to be used for storing data records, wherein the type of storage location comprises one of:

(a) magnetic disk-based storage, (b) solid state drive (SSD), (c) volatile RAM (random access memory), (d) non-volatile RAM, (e) a database management system, or (f) a storage node of a network-accessible storage service implemented by a provider network.

13. The method as recited in claim 10, wherein the data persistence policy comprises an indication of a first type of storage location to be used for a first partition of the particular data stream, and an indication of a different type of storage location to be used for a second partition of the particular data stream.

14. The method as recited in claim 10, wherein the data persistence policy selected by the client does not include an indication of a type of storage location to be used for storing data records, further comprising performing, by the one or more computing devices:

selecting, by one or more components of the stream management service, a type of storage location to be used for storing data records.

15. The method as recited in claim 14, wherein said selecting a type of storage location comprises selecting a first type of storage location for a first partition of the particular data stream, and selecting a different type of storage location for a second partition of the particular data stream.

16. The method as recited in claim 10, wherein the data persistence policy includes an indication of a target latency, further comprising performing, by the one or more computing devices:

selecting, by one or more components of the stream management service, a type of storage location to be used for storing data records based at least in part on the target latency.

17. The method as recited in claim 10, wherein the data persistence policy selected by the client comprises an indication of a requested data durability level, further comprising performing, by the one or more computing devices:

selecting a plurality of storage locations at which data records of the particular data stream are to be stored based at least in part on the requested data durability level.

18. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

determining a billing amount that a particular client is to be charged for stream management operations based at least in part on one or more of;

(a) a data ingestion policy selected by the particular client, and (b) a data persistence policy selected by the particular client.

19. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

determining the number of data ingestion nodes to be configured for the particular data stream based at least in part on the partitioning policy in accordance with which the particular data record is designated as a member of the particular partition based on one or more attributes of the particular data record.

20. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

determining a number of data storage nodes to be configured for the particular data stream based at least in part on a partitioning policy in accordance with which (a) the particular data record is designated as a member of a particular partition based on one or more attributes of the particular data record and (b) a data storage node of the number of data storage nodes is selected to store data records of the particular partition.

21. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

storing, corresponding to each data record of a plurality of data records of the particular data stream, including the particular data record, a respective sequence number usable to respond to read requests for ordered sets of data records;

receiving, from a submitter of a different data record of the particular data stream, an indication of a minimum sequence number to be stored corresponding to the different data record; and

storing, corresponding to the different data record, a particular sequence number that is greater than or equal to the minimum sequence number.

22. A non-transitory computer-accessible storage medium storing program instructions that, when executed on one or more processors, cause the one or more processors to:

implement a set of programmatic interfaces enabling a client of a network-based data stream management service to select, for a particular data stream, a data persistence policy from among a plurality of data persistence policies, wherein the plurality of data persistence policies includes (a) a multiple-replica persistence policy in accordance with which multiple copies of a data record of the particular data stream are to be stored at respective storage locations and (b) a single-replica persistence policy in which a single copy of a data record of the particular data stream is to be stored, wherein data records for the particular data stream are ingested into the particular data stream based on a selected data ingestion policy;

receive, via a programmatic interface of the set, a request indicating that the client has selected the multiple-replica persistence policy for the particular data stream;

determine a number of data storage nodes to be configured for the particular data stream based at least in part on a partitioning policy in accordance with which a data storage node of the number of data storage nodes is selected to store data records of a particular partition of the particular data stream; and

configure a plurality of storage nodes to implement the multiple-replica persistence policy for data records of the particular data stream.

23. The non-transitory computer-accessible storage medium as recited in claim 22, wherein at least one data persistence policy of the plurality of data persistence policies comprises an indication of a type of storage location to be used for storing data records, wherein the type of storage location comprises one of:

(a) magnetic disk-based storage, (b) solid state drive (SSD), (c) volatile RAM (random access memory), (d) non-volatile RAM, (e) a database management system, or (f) a storage node of a network-accessible storage service implemented by a provider network.

24. The non-transitory computer-accessible storage medium as recited in claim 22, wherein the multiple-replica persistence policy selected by the client comprises an indication of a requested data durability level, wherein the instructions, when executed on the one or more processors, cause the one or more processors to:

determine a number of storage nodes to be configured based at least in part on the requested data durability level.

25. The non-transitory computer-accessible storage medium as recited in claim 22, wherein the instructions, when executed on the one or more processors, cause the one or more processors to:

determine the number of data storage nodes to be configured for the particular data stream based at least in part on the partitioning policy in accordance with which a particular data record is designated as a member of the particular partition based on one or more attributes of the particular data record.

26. The non-transitory computer-accessible storage medium as recited in claim 22, wherein the instructions, when executed on the one or more processors, cause the one or more processors to:

implement a different set of programmatic interfaces enabling a different client of the network-based stream service to select, for a different data stream, a data persistence policy from among the plurality of data persistence policies;

receive, via a different programmatic interface of the set, a different request indicating that a different client of the network-based stream service has selected a different data persistence policy for a different data stream; and

configure the plurality of storage nodes to implement the different data persistence policy for data records of the different data stream.