Measuring usage of computing resources by storing usage events in a distributed file system

• US 10,725,979 B2
• Filed: 03/24/2017
• Issued: 07/28/2020
• Est. Priority Date: 03/24/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lin, Yao Huang, Westphal, Matthew, Tan, Yu, Shaji, Ullattil, Bandyopadhyay, Moinak
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US15/468,140
• Publication Number:
  US 20180276239A1
• Time in Patent Office: 1,222 Days
• Field of Search: 707827, 707690, 707688, 707699, 707692
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/184 implemented as replicated file system

Abstract

To measure usage of computing resources on a computer, a logging service running on the computer generates event data. The computer transmits the event data to an event processing system. With a large number of computers, each computer transmits its event data to the event processing system. The event processing system stores the event data in a storage system as a file system object, such as a stream or file, in which different data fields of the received event data are stored in a structured or semi-structured manner. The event data can be processed in parallel on different pivots using map-reduce operations. Such processing can include, but is not limited to, de-duplicating event data, aggregating event data related to a resource into measurements of usage of that resource, and grouping original data or aggregated data by user or group of user for further analysis and reporting.

Claims

1. A computer system comprising:

a processor; and

a computer storage device holding instructions executable by the processor to receive raw event data over a computer network from a plurality of computers, the raw event data including usage events, each usage event comprising a plurality of fields, including data indicating a time at which the usage event was captured, data indicating a computing resource, and data indicating a user or group of users;

wherein the computer storage device is further configured to hold a plurality of file system objects in a distributed file system, each file system object including the data from usage events received during a time period associated with the file system object, each file system object storing each field in the plurality of fields as a separate column in the file system object; and

wherein the computer storage device further holds instructions executable by the processor to;

receive summary information related to the raw event data, compare the summary information and the raw event data, based on such comparison, identify whether there is any discrepancy between the raw event data and the summary information related to the raw event data, and store summary report data indicating any such discrepancy in a summary file system object.

2. The computer system of claim 1, wherein the plurality of fields of a usage event further comprises:

a unique identifier of the usage event.

3. The computer system of claim 2, wherein the computer storage device is further configured to hold instructions executable by the processor to identify any usage events in one or more file system objects having a matching unique identifier, and to generate a file system object without duplicate usage events.

4. The computer system of claim 2, wherein the computer storage device is further configured to hold instructions executable by the processor to:

receive data indicating the identifiers of usage events processed by the computer system; and

compare the received data to the usage events stored in the file system objects to ensure that the usage events in the received data are stored in the file system objects.

5. The computer system of claim 1, wherein the computer storage device is further configured to hold instructions executable by the processor to:

identify file system objects corresponding to a first period of time;

identify usage events in the identified file system objects having a same resource identifier and a same user;

aggregate usage information associated with the identified usage events; and

create file system objects including the aggregated usage information.

6. The computer system of claim 1, wherein the fields of usage events include data about a resource including at least a resource identifier.

7. The computer system of claim 1, wherein the instructions are further executable by the processor to determine a typical distribution of usage events from the raw event data and the summary information, and to identify a discrepancy between the raw event data and the summary information responsive to an incoming usage event having a low probability in the typical distribution.

8. The computer system of claim 1, wherein the instructions are further executable by the processor to automatically output an alert responsive to identifying any discrepancy between the raw event data and the summary information related to the raw event data.

9. A computer-implemented process, comprising:

receiving raw event data over a computer network from a plurality of computers, the raw event data including usage events, each usage event comprising a plurality of fields, including data indicating a time at which the usage event was captured, data indicating a computing resource, and data indicating a user or group of users;

storing a plurality of file system objects in a distributed file system, each file system object including the data from usage events received during a time period associated with the file system object, each file system object storing each field in the plurality of fields as a separate column in the file system object;

receiving summary information related to the raw event data;

comparing the summary information and the raw event data, and based on such comparison, identifying whether there is any discrepancy between the raw event data and the summary information related to the raw event data; and

storing summary report data indicating any such discrepancy in a summary file system object.

10. The computer implemented process of claim 9, wherein the plurality of fields of a usage event further comprises:

a unique identifier of the usage event.

11. The computer-implemented process of claim 10, further comprising:

identifying any usage events in one or more file system objects having a matching unique identifier; and

generating a file system object without duplicate usage events.

12. The computer-implemented process of claim 9, further comprising:

identifying file system objects corresponding to a first period of time;

identifying usage events in the identified file system objects having a same resource identifier and a same user;

aggregating usage information associated with the identified usage events; and

creating file system objects including the aggregated usage information.

13. The computer-implemented process of claim 9, wherein the fields of usage events include data about a resource including at least a resource identifier.

14. The computer-implemented process of claim 9, further comprising:

determining a typical distribution of usage events from the raw event data and the summary information, and

identifying a discrepancy between the raw event data and the summary information responsive to an incoming usage event having a low probability in the typical distribution.

15. A computer system comprising:

a processor; and

a computer storage device holding instructions executable by the processor to receive a first stream comprising raw event data over a computer network from a plurality of computers, the raw event data including usage events, each usage event comprising a plurality of fields, including data indicating a time at which the usage event was captured, data indicating a computing resource, and data indicating a user or group of users;

wherein the computer storage device is further configured to hold a plurality of file system objects in a distributed file system, each file system object including the data from usage events received during a time period associated with the file system object, each file system object storing each field in the plurality of fields as a separate column in the file system object; and

wherein the computer storage device further holds instructions executable by the processor to;

receive a second stream comprising summary information related to the raw event data, compare the first stream and the second stream, based on such comparison, identify whether there is any discrepancy between the raw event data and the summary information related to the raw event data, and store summary report data indicating any such discrepancy in a summary file system object.

16. The computer system of claim 15, wherein the plurality of fields of a usage event further comprises:

a unique identifier of the usage event.

17. The computer system of claim 16, wherein the computer storage device further holds instructions executable by the processor to identify any usage events in one or more file system objects having a matching unique identifier, and to generate a file system object without duplicate usage events.

18. The computer system of claim 16, wherein the computer storage device further holds instructions executable by the processor to:

receive data indicating the identifiers of usage events processed by the computer system; and

compare the received data to the usage events stored in the file system objects to ensure that the usage events in the received data are stored in the file system objects.

19. The computer system of claim 15, wherein the computer storage device is further configured to hold instructions executable by the processor to:

identify file system objects corresponding to a first period of time;

identify usage events in the identified file system objects having a same resource identifier and a same user;

aggregate usage information associated with the identified usage events; and

create file system objects including the aggregated usage information.

20. The computer system of claim 15, wherein the fields of usage events include data about a resource including at least a resource identifier.