Large scale real-time multistaged analytic system using data contracts

• US 9,367,619 B2
• Filed: 07/29/2011
• Issued: 06/14/2016
• Est. Priority Date: 07/29/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Katiyar, Atul, Amradkar, Purushottam, Gupta, Nitin
• Primary Examiner(s): Hwa, Shyue Jiunn
• Application Number:
  US13/194,084
• Publication Number:
  US 20130031143A1
• Time in Patent Office: 1,782 Days
• Field of Search: 707/805
• US Class Current: 1/1
• CPC Class Codes: G06F 16/74 Browsing; Visualisation therefor end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic program guides, H04N21/472
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

An analytic system may have a number of processing stages. One or more data sources may provide data to a first processing stage. The first processing stage may specify one or more data contracts having a schema describing a layout and types of data provided by the one or more data sources. Each of the processing stages may specify a respective data contract having a schema such that the processing stages may understand a layout and types of data provided as input to the processing stages. The data contracts me further specify a valid range of values for various items of data described by schemas. Data not conforming to a data contract may be automatically filtered out such that a corresponding processing stage may not be provided with the non-conforming data.

Claims

1. A system comprising:

a receiver configured to receive heterogeneous input data that is provided by at least one of a plurality of data sources;

a memory configured to store at least one data contract that specifies a layout of the heterogeneous input data;

a processor configured to;

process, according to the layout specified in each of the at least one data contract, the heterogeneous input data without transforming the heterogeneous input data to conform to a generic schema;

filter, according to the at least one data contract, the heterogeneous input data that does not conform to the at least one data contract;

implement a collection processing stage configured to collect data provided by the at least one of a plurality of data sources;

implement an aggregation processing stage configured to process and aggregate data provided by the collection processing stage; and

implement a visualization processing stage configured to render data provided from the aggregation processing stage to an output device.

2. The system of claim 1, where the specified layout of each of the at least one data contract comprises a schema that describes the heterogeneous input data that is provided by the corresponding one of the plurality of data sources.

3. The system of claim 1 configured to provide, via the visualization processing stage, user interface building blocks.

4. A machine-readable storage medium storing instructions for more than a transitory time period that, based on execution by at least one processor of at least one computing device that includes memory, configure the at least one computing device to:

execute, by the at least one computing device, at least one of a plurality of processing stages, where each processing stage of the plurality of processing stages includes at least one data contract that specifies a layout of heterogeneous input data received by the each processing stage that is provided by at least one of a plurality of data sources that includes the plurality of processing stages, where a first of the processing stages is a collection processing stage configured to collect data provided by the at least one of a plurality of data sources, where a second of the processing stages is an aggregation processing stage configured to process and aggregate data provided from the collection processing stage, and where a third of the processing stages is a visualization processing stage configured to render data provided from the aggregation processing stage to an output device;

process, via the each processing stage according to the layout specified in each of the corresponding at least one data contract, the heterogeneous input data received by the each processing stage without transforming the received heterogeneous input data to conform to a generic schema; and

filter, via the each processing stage according to the corresponding at least one data contract, the heterogeneous input data received by the each processing stage that does not conform to the corresponding at least one data contract.

5. The machine-readable storage medium of claim 4, where the specified layout of each of the at least one data contract comprises a schema that describes the heterogeneous input data that is provided by the corresponding one of the plurality of data sources.

6. The machine-readable medium of claim 4, where the instructions further configure the at least one computing device to provide, via the visualization processing stage, user interface building blocks.

7. A method performed on at least one computing device that includes at least one processor and memory, the method comprising:

executing, by the at least one computing device, at least one of a plurality of processing stages, where each processing stage of the plurality of processing stages includes at least one data contract that specifies a layout of heterogeneous input data received by the each processing stage that is provided by at least one of a plurality of data sources that includes the plurality of processing stages, where a first of the processing stages is a collection processing stage configured to collect data provided by the at least one of a plurality of data sources, where a second of the processing stages is an aggregation processing stage configured to process and aggregate data provided by the collection processing stage, and where a third of the processing stages is a visualization processing stage configured to render data provided from the aggregation processing stage, the method further comprising using, by the visualization processing to an output device;

processing, via the each processing stage according to the layout specified in each of the corresponding at least one data contract, the heterogeneous input data received by the each processing stage without transforming the received heterogeneous input data to conform to a generic schema; and

filtering, via the each processing stage according to the corresponding at least one data contract, the heterogeneous input data received by the each processing stage that does not conform to the corresponding at least one data contract.

8. The method of claim 7, where the specified layout of each of the at least one data contract comprises a schema that describes the heterogeneous input data that is provided by the corresponding one of the plurality of data sources.

9. The method of claim 7 further comprising providing, via the visualization processing stage, user interface building blocks.