Partitioned batch processing for a usage analysis system

• US 10,706,073 B1
• Filed: 12/03/2014
• Issued: 07/07/2020
• Est. Priority Date: 12/03/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Tamkin, Preston Alan, Dunham, Phillip Lloyd, Khudoliy, Oleksiy
• Primary Examiner(s): Almani, Mohsen
• Application Number:
  US14/559,868
• Time in Patent Office: 2,043 Days
• Field of Search: 707609
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

A usage analysis service that obtains session events from application instances, updates usage metrics for the application instances according to the session events, and stores or updates aggregated usage metrics for each application according to the session events. Updates for the instance usage metrics may be partitioned according to instance identifier, collected in batches, and applied to respective instance records in an instances table by one or more instance update nodes. Updates for the aggregated usage metrics may be partitioned according to application identifier, collected in batches, processed to combine multiple updates, and applied to respective aggregate records in an application usage table by multiple application usage update nodes. The system may provide an interface that retrieves aggregated usage metrics from the application usage table and presents application usage information for the applications to consumers.

Claims

1. A system, comprising:

one or more memory or storage devices to implement a data store that stores an application usage table comprising a plurality of records, each record corresponding to a particular one of a plurality of applications and including one or more aggregated usage metrics for the respective application;

one or more computing devices, comprising one or more hardware processors and associated memory, configured to implement one or more session event nodes configured to;

generate updates for the records in the application usage table according to session events received from a plurality of devices of a plurality of users, each session event indicating new usage data for a respective application instance on a respective device, and each update including an application identifier indicating one of the plurality of applications and increments for the one or more aggregated usage metrics of the respective application, wherein, for at least one of the plurality of applications, the session events received from two or more different devices of two or more different users indicate new usage data for two or more different application instances of the same application, and wherein the updates are generated as the session events are received; and

partition the updates into a plurality of partitions according to the application identifier;

a plurality of computing devices, each comprising one or more hardware processors and associated memory, configured to implement a plurality of application usage update nodes distinct from the one or more session event nodes, each application usage update node associated with a different one of the plurality of partitions, and responsible for updating a different one of the records in the application usage table according to the updates from its associated partition, and each application usage update node configured to;

obtain the updates from the respective different one of the plurality of partitions; and

apply the updates to the respective different one of the plurality of records in the application usage table corresponding to the respective particular one of the plurality of applications;

wherein the updates obtained and applied by at least one of the plurality of application usage update nodes to the respective different one of the plurality of records corresponds to session events received, by one or more of the session event nodes, from two or more different devices of two or more different users.

2. The system as recited in claim 1, wherein each application usage update node is further configured to combine two or more of the updates obtained from the respective partition to generate a combined update, and wherein, to apply the updates to the respective record in the application usage table, the application usage update node applies the combined update to the respective record in the application usage table.

3. The system as recited in claim 1, wherein the plurality of application usage update nodes are configured to perform said obtaining and said applying according to a batch update interval.

4. The system as recited in claim 1, wherein the data store further stores an instances table comprising a plurality of instance records, each instance record corresponding to an instance of one of the plurality of applications on one of the plurality of devices and including one or more usage metrics for the respective application instance on the respective device, and wherein the session event nodes are further configured to update the instance records in the instances table according to the received session events.

5. The system as recited in claim 4, wherein, to update the instance records in the instances table according to the received session events, the session event nodes are configured to:

generate instance updates according to the session events, each instance update including an application instance identifier indicating a particular application instance and updates for one or more instance usage metrics of the respective application instance; and

partition the instance updates into a plurality of instance update partitions according to the application instance identifier;

wherein the system further comprises one or more computing devices configured to implement a plurality of instance update nodes, each instance update node corresponding to one of the instance records in the instances table, and each instance update node configured to;

obtain the instance updates from a respective one of the plurality of instance update partitions; and

apply the instance updates to a respective one of the instance records in the instances table.

6. The system as recited in claim 1, wherein the data store is provided by an database service, wherein each application usage update node is configured to apply the updates to the respective record in the application usage table via an application programming interface (API) of the database service.

7. The system as recited in claim 1,

wherein, to partition the updates into a plurality of partitions according to the application identifier, the one or more session event nodes are configured to send the updates to a streaming service via an application programming interface (API) of the streaming service, wherein the streaming service puts the updates into the respective partitions using the application identifier in the updates as a partition key;

and

wherein, to obtain the updates from a respective one of the plurality of partitions, each application usage update node is configured to read the updates from the respective partition via the API of the streaming service.

8. The system as recited in claim 1, further comprising one or more computing devices configured to implement a consumer interface for data stored in the application usage table, wherein the consumer interface is configured to:

receive requests for application usage information from one or more usage information consumers;

retrieve aggregated usage metrics from one or more of the records in the application usage table according to the requests;

generate the requested application usage information at least in part according to the retrieved aggregated usage metrics; and

provide the requested application usage information to respective ones of the one or more usage information consumers.

9. A method, comprising:

generating, by one or more session event nodes implemented on one or more computing devices, updates for a plurality of application usage records according to session events received from a plurality of devices of a plurality of users, each session event indicating new usage data for a respective application instance on a respective device of a respective user, and each update including an application identifier indicating one of a plurality of applications and increments for one or more aggregated usage metrics of the respective application, wherein, for at least one of the plurality of applications, the one or more session event nodes receives a plurality of session events from two or more different devices of two or more different users indicating new usage data for two or more different application instances of the same application, and wherein the updates are generated as the session events are received;

partitioning, by the one or more session event nodes, the updates into a plurality of partitions according to the application identifier; and

applying, by a plurality of application usage update nodes distinct from the one or more session event nodes, batches of the updates from the partitions to respective ones of the plurality of application usage records, each application usage record corresponding to a particular one of the plurality of applications and including one or more aggregated usage metrics for the respective application, wherein each application usage update node is associated with a different partition of the plurality of partitions, and applies updates from the different partition of the plurality of partitions to a different one of the plurality of application usage records;

wherein the updates obtained and applied by at least one of the plurality of application usage update nodes to the respective different one of the plurality of application usage records corresponds to session events received, by one or more of the session event nodes, from two or more different devices of two or more different users.

10. The method as recited in claim 9, further comprising combining two or more of the updates in a respective partition to generate a combined update, and wherein said applying batches of the updates from the partitions to respective ones of a plurality of application usage records comprises applying the combined updates to the respective records.

11. The method as recited in claim 9, wherein said applying batches of the updates from the partitions to respective ones of a plurality of application usage records is performed according to a batch update interval.

12. The method as recited in claim 9, further comprising updating a plurality of instance records according to the received session events, wherein each instance record corresponds to an instance of one of the plurality of applications on one of the plurality of devices and includes one or more usage metrics for the respective application instance on the respective device.

13. The method as recited in claim 12, wherein said updating a plurality of instance records according to the received session events comprises:

generating instance updates according to the session events, each instance update including an application instance identifier indicating a particular application instance and updates for one or more instance usage metrics of the respective application instance;

partitioning the instance updates into a plurality of instance update partitions according to the application instance identifier; and

applying the instance updates from the instance update partitions to respective ones of the instance records.

14. The method as recited in claim 13, wherein said applying the instance updates from the instance update partitions to respective ones of the instance records is performed according to a batch update interval.

15. The method as recited in claim 9, wherein said partitioning the updates into a plurality of partitions according to the application identifier comprises sending the updates to a streaming service that places the updates into the respective partitions using the application identifier as a partition key.

16. The method as recited in claim 15, wherein said applying batches of the updates from the partitions to respective ones of a plurality of application usage records comprises reading the updates from the respective partitions via the API of the streaming service.

17. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a usage analysis system comprising:

one or more session event nodes configured to;

generate updates for a plurality of records in an application usage table according to session events received from a plurality of devices of a plurality of users, each session event indicating new usage data for a respective application instance on a respective device of a respective user, and each update including an application identifier indicating one of a plurality of applications and increments for one or more aggregated usage metrics of the respective application, wherein, for at least one of the plurality of applications, the session events from two or more different devices of two or more different users indicate new usage data for two or more different application instances of the same application, and wherein the updates are generated as the session events are received; and

partition the updates into a plurality of partitions according to the application identifier;

a plurality of application usage update nodes distinct from the one or more session event nodes configured to;

obtain the updates from respective ones of the plurality of partitions; and

apply the updates to respective ones of the plurality of records in the application usage table, each record in the application usage table corresponding to a particular one of the plurality of applications and including one or more aggregated usage metrics for the respective application, wherein each application usage update node is associated with a different partition of the plurality of partitions, and applies updates from the different partition to a different record of the plurality of records;

wherein the updates obtained and applied by at least one of the plurality of application usage update nodes to the respective different one of the plurality of records corresponds to session events received, by one or more of the session event nodes, from two or more different devices of two or more different users.

18. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the one or more session event nodes are further configured to:

generate instance updates according to the session events, each instance update including an application instance identifier indicating a particular application instance and updates for one or more instance usage metrics of the respective application instance; and

partition the instance updates into a plurality of instance update partitions according to the application instance identifier;

wherein the usage analysis system further comprises a plurality of instance update nodes configured to;

obtain the instance updates from respective ones of the plurality of instance update partitions; and

apply the instance updates to respective ones of a plurality of instance records in an instances table, each instance record in the instances table corresponding to an instance of one of the plurality of applications on one of the plurality of devices and including one or more usage metrics for the respective application instance on the respective device.

19. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the application usage update nodes are further configured to combine updates obtained from the partitions to generate combined updates, and wherein, to apply the updates to respective ones of a plurality of records in an application usage table, the application usage update nodes apply the combined updates to the respective records in the application usage table.

20. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the plurality of application usage update nodes are configured to perform said obtaining and said applying according to a batch update interval.