Dynamically processing an event using an extensible data model

• US 8,874,526 B2
• Filed: 09/08/2010
• Issued: 10/28/2014
• Est. Priority Date: 03/31/2010
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Hsieh, Jonathan Ming-Cyn, Robinson, Henry Noel
• Primary Examiner(s): To, Baoquoc N
• Application Number:
  US12/877,930
• Publication Number:
  US 20110246528A1
• Time in Patent Office: 1,511 Days
• Field of Search: 707/672
• US Class Current: 707/672
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

Systems and methods of dynamically processing an event using an extensible data model are disclosed. One embodiment includes, specifying attributes of the event in a data model; the data model being extensible to add properties to the event as the dataset is streamed from the source to the sink.

Claims

1. A method for dynamically processing an event including a dataset that is streamed from a source to a sink via nodes, the method comprising:

recording the event including the dataset in a data model;

extracting a timestamp from raw data in the dataset;

specifying, based on the timestamp, a priority of the event in a priority field included in the data model;

annotating, based on the priority, the event in the data model with an attribute in a metadata table included in the data model, wherein the attribute includes a map that directs how the event is to be streamed to a subsequent node,

wherein the metadata table included in the data model is extensible to add additional attributes to the event by subsequent nodes which are configured to further process the dataset as the event is streamed from the source to the sink.

2. The method of claim 1, further comprising:

appending additional data fields to the data model in response to additional sources being added to the event.

3. The method of claim 1, wherein, the priority is specified by a logging system.

4. The method of claim 1, wherein, the data model further includes a source machine field.

5. The method of claim 1, wherein, the data model further includes a body.

6. The method of claim 1, wherein, the data model includes a timestamp.

7. The method of claim 6, further comprising, using the timestamp to consolidate the event by bucketing the dataset with other events.

8. The method of claim 1, further comprising, grouping the dataset based on project identifier or person.

9. The method of claim 1, further comprising:

consolidating the event by bucketing the event with other events based on the attributes.

10. A method for dynamically processing an event including a dataset that is streamed from a source to a sink via nodes, the method comprising:

recording the event including the dataset in a data model;

extracting a timestamp from raw data in the dataset;

specifying, based on the timestamp, a priority of the event in a priority field included in the data model;

annotating, based on the priority, the event in the data model with an attribute in a metadata table included in the data model, wherein the attribute includes a map that directs how the event is to be streamed to a subsequent node; and

consolidating the event by bucketing the event with other events based on the key-value data;

wherein the metadata table included in the data model is extensible to add additional attributes to the event by subsequent nodes which are configured to process the dataset as the event is streamed from the source to the sink.

11. The method of claim 10, wherein, the metadata table is used to specify routing of the event to multiple or different storage locations.

12. The method of claim 10, wherein, the metadata table is used to specify outputting of the dataset at the sink in multiple or different formats.

13. The method of claim 10, wherein, the analytic features include feature extraction.

14. The method of claim 10, wherein, the analytic features include filtering.

15. The method of claim 10, wherein, the analytic features include transformation.

16. A system for dynamically processing an event including a dataset that is streamed from a source to a sink via nodes, the method, comprising:

means for recording the event including the dataset in a data model;

means for extracting a timestamp from raw data in the dataset;

means for specifying, based on the timestamp, a priority of the event in a priority field included in the data model;

wherein the data model includes a metadata table that is extensible to add additional attributes to the event by subsequent nodes which are configured to further process the dataset as the event is streamed from the source to the sink; and

means for annotating, based on the priority, the event in the data model with an attribute in the metadata table, wherein the attribute includes a map that directs how the event is to be streamed to a subsequent node.

17. The system of claim 16, wherein, the metadata table is used to specify routing of the event to multiple or different storage locations.

18. The system of claim 16, wherein, the metadata table is used to specify outputting of the dataset at the sink in multiple or different formats.