STREAM PROCESSING IN SEARCH DATA PIPELINES

• US 20160224632A1
• Filed: 02/02/2015
• Published: 08/04/2016
• Est. Priority Date: 02/02/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lu, Wei, Kinoti, Michael, Zhang, Xian, Zou, Guixi, He, Yin, Rius, Xavier Drudis, Yan, Shengquan, Yu, Peng, Rosenberg, Miriam, Zheng, Zijian
• Granted Patent: US 10,628,423 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

Architecture that decomposes of one or more monolithic data concepts into atomic concepts and related atomic concept dependencies, and provides streaming data processing that processes individual or separate (atomic) data concepts and defined atomic dependencies. The architecture can comprise data-driven data processing that enables the plug-in of new data concepts with minimal effort. Efficient processing of the data concepts is enabled by streaming only required data concepts and corresponding dependencies and enablement of the seamless configuration of data processing between stream processing systems and batch processing systems as a result of data concept decomposition. Incremental and non-incremental metric processing enables realtime access and monitoring of operational parameters and queries.

Claims

1. A system, comprising:

a separation component configured to separate a monolithic data concept into individual concepts and generate concept dependencies for the individual concepts;

a selection component configured to select an individual concept and associated concept dependency for stream processing;

a stream processing component configured to perform stream processing of the selected individual concept and the associated dependency; and

a processor configured to execute instructions in a memory, the instructions executed to enable the separation component, the selection component, and the stream processing component.

2. The system of claim 1, wherein the stream processing component is configured to stream process the selected individual concept and the associated dependency as part of data driven data processing.

3. The system of claim 1, wherein the selection component is configured to select only a specific individual concept and its associated dependency for a data driven requirement.

4. The system of claim 1, further comprising a metric component configured to compute a metric of a data concept during the stream processing, based on availability of the data concept, and store the metric in a storage.

5. The system of claim 4, wherein the metric component is configured to update the metric over time as a new data concept becomes available.

6. The system of claim 1, wherein the metric component is configured to compute a partial value of the metric during the stream processing of the data concept and associated concept dependency.

7. The system of claim 1, wherein the metric component is configured to access a value of a metric during the stream processing, the value is a latest value of the metric.

8. The system of claim 1, wherein the stream processing component is configured to enable communication with a batch processing component to communicate the concept and associated dependency.

9. A method, comprising acts of:

decomposing a monolithic data concept into individual instances of both concepts and associated concept dependencies;

selecting one or more of the instances of both concepts and associated concept dependencies for stream processing; and

stream processing the selected instances of the individual concepts and the associated dependencies on demand.

10. The method of claim 9, further comprising decomposing the monolithic data as part of a data driven process.

11. The method of claim 9, further comprising stream processing only a specific individual concept and its associated dependencies for a data driven requirement.

12. The method of claim 9, further comprising computing metrics during the stream processing, which metrics represent availability of the data concepts, and storing the metrics in a storage.

13. The method of claim 12, further comprising updating the metrics over time, as new data concepts become available.

14. The method of claim 12, further comprising computing a partial value of a metric during the stream processing of the data concepts and associated dependencies.

15. The method of claim 12, further comprising accessing a value of a subset of the metrics during the stream processing to obtain a latest value of the subset of metrics.

16. The method of claim 9, further comprising interfacing the stream processing to batch processing based on the individual instances of both concepts and associated concept dependencies.

17. A method, comprising acts of:

decomposing a monolithic data concept into individual instances of both concepts and associated concept dependencies;

selecting one or more of the instances of both concepts and associated concept dependencies for stream processing;

stream processing the selected instances of the individual concepts and the associated dependencies as part of a data driven data process; and

computing a partial value of a metric during the stream processing of the data concepts and associated dependencies.

18. The method of claim 17, further comprising computing the partial value of the metric incrementally or non-incrementally.

19. The method of claim 17, further comprising processing a realtime query against the stream processed selected instances of the individual concepts and the associated dependencies to return a query result of events.

20. The method of claim 17, further comprising selecting only an instance of a concept and associated dependency based on a data driven requirement.