LARGE SCALE REAL-TIME MULTISTAGED ANALYTIC SYSTEM USING DATA CONTRACTS

• US 20130031143A1
• Filed: 07/29/2011
• Published: 01/31/2013
• Est. Priority Date: 07/29/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Katiyar, Atul, Amradkar, Purushottam, Gupta, Nitin
• Granted Patent: US 9,367,619 B2
• Time in Patent Office: Days
• US Class Current: 707/805
• CPC Class Codes: G06F 16/74 Browsing; Visualisation therefor end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic program guides, H04N21/472
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

An analytic system may have a number of processing stages. One or more data sources may provide data to a first processing stage. The first processing stage may specify one or more data contracts having a schema describing a layout and types of data provided by the one or more data sources. Each of the processing stages may specify a respective data contract having a schema such that the processing stages may understand a layout and types of data provided as input to the processing stages. The data contracts me further specify a valid range of values for various items of data described by schemas. Data not conforming to a data contract may be automatically filtered out such that a corresponding processing stage may not be provided with the non-conforming data.

Claims

1. At least one computing device implementing a large-scale real-time multistaged analytic system, each of the at least one computing device comprising:

at least processor; and

a memory operatively connected with the at least one processor, the memory having instructions for the at least one processor to perform a method comprising;

executing a plurality of processing stages of a pipeline for analyzing data, each processing stage of the plurality of processing stages using a respective data contract that specifies a description of inter-stage data to be exchanged across a respective processing stage boundary of the each processing stage, such that the each processing stage is completely agnostic with respect to a data schema and semantics of the inter-stage data consumed by the each processing stage, wherein

when a change occurs with respect to data provided to a processing stage of the pipeline, in order to accommodate the change, a data contract that specifies a description of the data to be provided to the processing stage is changed and processing logic of the processing stage of the pipeline remains unchanged.

2. The at least one computing device of claim 1, wherein:

a first processing stage of the pipeline uses the respective data contract with respect to each of at least one source of data to be consumed by the first processing stage; and

each remaining stage of the plurality of processing stages of the pipeline uses the respective data contract describing the inter-stage data provided by a respective previous processing stage in order for the each remaining stage of the plurality of processing stages of the pipeline to consume the inter-stage data from the respective previous processing stage.

3. The at least one computing device of claim 1, wherein at least one data contract of the respective data contracts is specified such that bad data or data of no interest to the one or more processing stages associated with the at least one data contract are filtered out of the inter-stage data being provided to the one or more processing stages.

4. The at least one computing device of claim 1, wherein a first processing stage of the plurality of processing stages is a collection processing stage for collecting data provided from at least one data source.

5. The at least one computing device of claim 4, wherein:

a second processing stage of the plurality of processing stages is an aggregation processing stage for processing data provided from the collection processing stage and aggregating the processed data, and

the aggregation processing stage uses the respective data contract that specifies a schema of the data provided by the collection processing stage in order to consume the data from the collection processing stage.

6. The at least one computing device of claim 5, wherein:

a third processing stage of the plurality of processing stages is a visualization processing stage for rendering data provided to the visualization processing stage according to the respective data contract conforming to data provided to the visualization processing stage which was processed by the aggregation processing stage, the visualization processing stage binding at least some of the data provided to the visualization processing stage for rendering to an output device.

7. The at least one computing device of claim 6, wherein the visualization processing stage provides basic user interface building blocks that render data based on the at least some of the data bound by the visualization processing stage.

8. A machine-readable storage medium for storing data, instructions, or other information for more than a transitory time period, the machine-readable storage medium having instructions recorded thereon for at least one processor of a computing device, when the at least one processor executes the instructions, the computing device performs a method comprising:

executing an analytic system having a plurality of processing stages of a pipeline, each processing stage of the plurality of processing stages having a respective data contract with a corresponding immediately adjacent processing stage within the pipeline, the respective data contract describing inter-stage data provided by either the corresponding immediately adjacent processing stage to the each processing stage or from the each processing stage to the corresponding immediately adjacent processing stage, the each processing stage being completely agnostic with respect to a data schema and semantics of the inter-stage data, wherein

when a change occurs with respect to data provided to a processing stage of the pipeline, in order to accommodate the change, a data contract that specifies a description of the data to be provided to the processing stage is changed and processing logic of the processing stage of the pipeline remains unchanged.

9. The machine-readable storage medium of claim 8, wherein the method further comprises:

rendering, by one processing stage of the plurality of processing stages, the inter-stage data based on the respective data contract between the one processing stage and a previous immediately adjacent processing stage within the pipeline.

10. The machine-readable storage medium of claim 8, wherein one processing stage of the plurality of processing stages comprises a plurality of user interface building blocks, each user interface building block being for rendering the inter-stage from a previous immediately adjacent processing stage within the pipeline based on the respective data contract between the one processing stage and the previous immediately adjacent processing stage.

11. The machine-readable storage medium of claim 8, wherein the data contract between two of the plurality of processing stages filters out any data that does not conform to the data contract such that only data that conforms to the data contract is provided to one of the two of the plurality of processing stages that consumes the data.

12. The machine-readable storage medium of claim 8, wherein when an instrumentation methodology of a data source changes, performing only a reconfiguration of at least one of the respective data contracts results in an analytic system being capable of working with data conforming to the changed instrumentation methodology.

13. The machine-readable medium of claim 8, wherein:

a first processing stage of the plurality of processing stages is a collection processing stage for collecting data from at least one data source;

a second processing stage is an aggregation processing stage for consuming and processing data from the collection processing stage; and

a third processing stage is a visualization processing stage for consuming and rendering data from the aggregation processing stage.

14. A machine-implemented method for analyzing data provided from at least one data source, the machine-implemented method comprising:

executing an analytic system having a plurality of processing stages of a pipeline, each processing stage of the plurality of processing stages having a respective data contract with a corresponding immediately adjacent processing stage within the pipeline, the respective data contract specifying a data schema for inter-stage data produced either from the corresponding immediately adjacent processing stage to the each processing stage or from the each processing stage to the corresponding immediately adjacent processing stage, the each processing stage being completely agnostic with respect to the data schema and semantics of the inter-stage data, wherein;

the machine-implemented method is implemented by at least one computing device, and

when a change occurs with respect to data provided to a processing stage of the pipeline, in order to accommodate the change, a data contract that specifies a description of the data to be provided to the processing stage is changed and processing logic of the processing stage of the pipeline remains unchanged.

15. The machine-implemented method of claim 14, wherein the analytic system is a web-analytic system.

16. The machine-implemented method of claim 14, further comprising:

filtering out data not conforming to any one of the respective data contracts such that each processing stage of the plurality of processing stages that consumes data based on the any one of the respective data contracts is provided with only data conforming to the any one of the data contracts.

17. The machine-implemented method of claim 14, wherein:

one of the respective data contracts specifies a range of values for an item of data, and the machine-implemented method further comprises;

filtering out ones of the items of data outside of the specified range of values such that a processing stage of the plurality of processing stages that consumes the items of data based on the one of the respective data contracts is provided with only the items of the data that are within the range of values specified by the one of the respective data contracts.

18. The machine-implemented method of claim 14, wherein application instrumentation is decoupled from ones of the processing stages that consume data generated through the application instrumentation.

19. The machine-implemented method of claim 14, wherein:

one of the plurality of processing stages is a collection processing stage that uses one of the respective data contracts specifying a data schema of data provided from a data source and consumed by the collection processing stage.

20. The machine-readable medium of claim 14, wherein:

one of the plurality of processing stages is a visualization processing stage that provides basic user interface building blocks that render data based on data bound by the visualization processing stage.