SCALABLE DATA FEED SYSTEM

• US 20170171337A1
• Filed: 01/30/2017
• Published: 06/15/2017
• Est. Priority Date: 02/20/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Garg, Manas, Bheemarajaiah, Anand, Prabhu, Sandeep, Srinivasan, Krishnan
• Granted Patent: US 10,673,969 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 12/1859 adapted to provide push services, e.g. data channels
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/55 Push-based network services
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A scalable data feed system is disclosed. The data feed system may be offered as a cloud service that can serve many enterprises or tenants that require data to be pulled from information sources such as FTP, POP3, databases, line of business systems, a topic subscription, or an RSS feed, and pushed the data to information sinks, such as SMTP, email, FTP, mobile phones, and other devices and services. A pull agent pumps data from pull sources and pushes the data out to push agent counterparts. The push agent transforms and sends the data in messages to push sink, such as FTP, SMTP, or a mobile device. Both the pull agent and the push agent services are implemented as replicated services over a structured overlay network for high availability that can serve multiple requests to pump out data from multiple pull sources and push the data to multiple information sinks.

Claims

1. A system for pulling data from a plurality of data sources, the system comprising:

a server configured to provide a pull service for retrieving data from the plurality of data sources, the server further configured to;

retrieve information identifying an assigned data source of the plurality of data sources, wherein the pull service is tasked with pulling data from the assigned data source;

receive a command to pull requested data;

schedule an attempt to pull the requested data from the assigned data source, wherein the schedule is maintained at least in part as a queue of work items, wherein work items specify the requested data to be pulled from the assigned data source;

execute a scheduled work item; and

if execution of the work item is successful, forward the requested data to a push service.

2. The system of claim 1, wherein the server is further configured to:

determine whether the requested data is maintained in a data source that is assigned to the pull service;

if the data source of the requested data is not assigned to the pull service;

identify a capable pull service that is assigned to the data source; and

forward the request to the capable pull service.

3. The system of claim 2, wherein the pull service assigned to the data source of the requested data is determined by performing an address resolution based on the address of the data source of requested data.

4. The system of claim 1, wherein the server is further configured to:

accept an additional assignment for pulling data from an additional assigned data source of the plurality of data sources, wherein the assignment of the additional data sink serves to additionally task the push service with pushing incoming data to the additional assigned data sink, and wherein the additional assignment results from adjustments to a partitioning of the plurality of data sinks among a plurality of push service instances.

5. The system of claim 1, wherein the server is further configured to:

identify any errors resulting from attempts to complete a scheduled work item; and

if execution of the work item is not successful, reschedule an additional attempt to pull the requested data from the assigned data source, wherein the additional attempt is rescheduled at least in part based on any identified errors.

6. The system of claim 1, wherein the command to pull requested data specifies an information sink that has been configured to accept the requested data.

7. The system of claim 1, wherein the push service is identified by performing an address resolution based on the address of the information sink.

8. The system of claim 1, wherein the server is further configured to:

transform the requested data to according to a protocol that is supported by the push service.

19. The method of claim 1, wherein the push agent service is a collection of push endpoint hosting services that push data to external information sinks.

20. The computer system of claim 19, wherein the collection of push endpoint hosting services partition the information sinks.

9. A method for pulling data from a plurality of data sources using a pull service, the method comprising:

retrieving information identifying an assigned data source of the plurality of data sources, wherein the pull service is tasked with pulling data from the assigned data source;

receiving a command to pull requested data;

scheduling an attempt to pull the requested data from the assigned data source, wherein the schedule is maintained at least in part as a queue of work items, wherein work items specify the requested data to be pulled from the assigned data source;

executing a scheduled work item; and

if the execution of the work item is successful, forwarding the requested data to a push service.

10. The method of claim 9, the method further comprising:

determining whether the requested data is maintained in a data source that is assigned to the pull service;

if the data source of the requested data is not assigned to the pull service;

identifying a capable pull service that is assigned to the data source; and

forwarding the request to the capable pull service.

11. The method of claim 10, wherein the pull service assigned to the data source of the requested data is determined by performing an address resolution based on the address of the data source of requested data.

12. The method of claim 9, the method further comprising:

accepting an additional assignment for pulling data from an additional assigned data source of the plurality of data sources, wherein the push service is tasked with pushing incoming data to the additional assigned data sink, and wherein the additional assignment results from adjustments to a partitioning of the plurality of data sinks among a plurality of push service instances.

13. The method of claim 9, the method further comprising:

identifying any errors resulting from execution of a scheduled work item; and

if an attempt to complete a work item is not successful, rescheduling an additional attempt to pull the requested data from the assigned data source, wherein the additional attempt is rescheduled at least in part based on any identified errors.

14. The method of claim 9, wherein the command to pull requested data specifies an information sink that has been configured to accept the requested data.

15. The method of claim 9, the method further comprising:

transforming the requested data to according to a protocol that is supported by the push service.

16. A method, comprising:

receiving data using a pull agent service that pulls data from an endpoint source;

sending the data from the pull agent service to a push agent service;

transforming the data at the push agent service; and

sending the transformed data as messages to an information sink.

17. The method of claim 16, wherein the pull agent service is a collection of pull endpoint hosting services that pull data from external information sources.

18. The method of claim 17, wherein the external information sources are partitioned among the collection of pull endpoint hosting services.