SCALABLE DATA FEED SYSTEM

• US 20130219009A1
• Filed: 02/20/2012
• Published: 08/22/2013
• Est. Priority Date: 02/20/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bheemarajaiah, Anand, Garg, Manas, Prabhu, Sandeep, Srinivasan, Krishnan
• Granted Patent: US 9,596,127 B2
• Time in Patent Office: Days
• US Class Current: 709/217
• CPC Class Codes: H04L 12/1859 adapted to provide push services, e.g. data channels
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/55 Push-based network services
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A scalable data feed system is disclosed. The data feed system may be offered as a cloud service that can serve many enterprises or tenants that require data to be pulled from information sources such as FTP, POP3, databases, line of business systems, a topic subscription, or an RSS feed, and pushed the data to information sinks, such as SMTP, email, FTP, mobile phones, and other devices and services. A pull agent pumps data from pull sources and pushes the data out to push agent counterparts. The push agent transforms and sends the data in messages to push sink, such as FTP, SMTP, or a mobile device. Both the pull agent and the push agent services are implemented as replicated services over a structured overlay network for high availability that can serve multiple requests to pump out data from multiple pull sources and push the data to multiple information sinks

Claims

1. A method, comprising:

receiving data using a pull agent service that pulls data from an endpoint source;

sending the data from the pull agent service to a push agent service;

transforming the data at the push agent service; and

sending the transformed data as messages to an information sink.

2. The method of claim 1, wherein the pull agent service and the push agent services are implemented as replicated services over a structured overlay network.

3. The method of claim 1, wherein the pull agent service is a collection of pull endpoint hosting services that pull data from external information sources.

4. The method of claim 3, wherein the external information sources are partitioned among the collection of pull endpoint hosting services.

5. The method of claim 1, wherein the push agent service is a collection of push endpoint hosting services that push data to external information sinks

6. The computer system of claim 5, wherein the collection of push endpoint hosting services partition the information sinks

7. The method of claim 1, further comprising:

scheduling a pull job from the endpoint source; and

reading the data as a stream from the endpoint source by the pull agent service.

8. The method of claim 7, further comprising:

converting the stream of data to an internal SOAP based protocol; and

pushing the converted stream of data to the push agent service.

9. The method of claim 1, further comprising:

performing an address resolution based on the address of the information sink to identify a partitioned push service that is responsible for the information sink.

10. The method of claim 9, further comprising:

transforming data pulled from an endpoint source to meet a protocol associated with the information sink.

11. The method of claim 10, further comprising:

publishing the protocol is published by the push agent service.

12. The method of claim 1, further comprising:

defining a message contract for each new information sink; and

detecting the message contract based upon an endpoint configuration.

13. The method of claim 1, further comprising:

transforming a downloaded data stream by a pull endpoint hosting service into a push agent protocol.

14. The method of claim 1, further comprising:

translating incoming messages at a push endpoint hosting service to a format specific to the information sink or at a pipeline transform component to a format configured by an end user.

15. A computer system, comprising:

one or more processors;

system memory;

one or more computer-readable storage media having stored thereon computer-executable instructions that, when executed by the one or more processors, causes the processors to perform a method for feeding data, the processor operating to;

download data from sources using a pull agent service;

push the data from the pull agent service to a push agent service;

transforming the data at the push agent service; and

sending the transformed data to an external destination.

16. The computer system of claim 15, wherein the pull agent service and the push agent services are implemented as replicated services over a structured overlay network for high availability.

17. The computer system of claim 15, wherein the pull agent service is a collection of pull endpoint hosting services and the sources are external information sources that are partitioned among the pull endpoint hosting services.

18. The computer system of claim 15, wherein the push agent service is a collection of push endpoint hosting services and the external destination is one of a plurality of information sinks that are partitioned among the push endpoint hosting services.

19. The computer system of claim 15, wherein a pull endpoint hosting service on the pull agent service transforms the downloaded data into a push agent protocol;

and wherein a push endpoint hosting service receives messages from the pull endpoint hosting service and translates the message to a format specific to the external destination.

20. A computer-readable storage medium storing computer-executable instructions that when executed by at least one processor cause the at least one processor to perform a method for feeding data, the method comprising:

downloading data from sources using a pull agent service;

pushing the downloaded data from the pull agent service to a push agent service;

transforming a protocol of the data at the push agent service; and

sending the transformed data to an information sink.