Scalable data feed system

• US 10,673,969 B2
• Filed: 01/30/2017
• Issued: 06/02/2020
• Est. Priority Date: 02/20/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bheemarajaiah, Anand, Garg, Manas, Prabhu, Sandeep, Srinivasan, Krishnan
• Primary Examiner(s): Lazaro, David R
• Application Number:
  US15/420,082
• Publication Number:
  US 20170171337A1
• Time in Patent Office: 1,219 Days
• CPC Class Codes: H04L 12/1859 adapted to provide push services, e.g. data channels
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/55 Push-based network services
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A scalable data feed system is disclosed. The data feed system may be offered as a cloud service that can serve many enterprises or tenants that require data to be pulled from information sources such as FTP, POP3, databases, line of business systems, a topic subscription, or an RSS feed, and pushed the data to information sinks, such as SMTP, email, FTP, mobile phones, and other devices and services. A pull agent pumps data from pull sources and pushes the data out to push agent counterparts. The push agent transforms and sends the data in messages to push sink, such as FTP, SMTP, or a mobile device. Both the pull agent and the push agent services are implemented as replicated services over a structured overlay network for high availability that can serve multiple requests to pump out data from multiple pull sources and push the data to multiple information sinks.

Claims

1. A system for pulling data from a plurality of data sources using a pull service, the system comprising:

one or more processors;

a memory in electronic communication with the one or more processors; and

instructions stored in the memory, the instructions being executable by the one or more processors to;

retrieve information identifying an assigned data source of the plurality of data sources, wherein the pull service is tasked with pulling data from the assigned data source;

receive a command to pull requested data;

schedule an attempt to pull the requested data from the assigned data source, wherein the schedule is maintained at least in part as a queue of work items, wherein work items specify the requested data to be pulled from the assigned data source;

execute a scheduled work item; and

if execution of the scheduled work item is successful, forward the requested data to a push service.

2. The system of claim 1, wherein the instructions are further executable by the one or more processors to:

determine whether the requested data is maintained in a data source that is assigned to the pull service; and

if the data source of the requested data is not assigned to the pull service;

identify a capable pull service that is assigned to the data source; and

forward the command to the capable pull service.

3. The system of claim 2, wherein the pull service assigned to the data source of the requested data is determined by performing an address resolution based on an address of the data source of requested data.

4. The system of claim 1, wherein the instructions are further executable by the one or more processors to:

accept an additional assignment for pulling data from an additional assigned data source of the plurality of data sources, wherein the additional assignment of the additional assigned data source serves to additionally task the push service with pushing incoming data to an additional assigned data sink, and wherein the additional assignment results from adjustments to a partitioning of a plurality of data sinks among a plurality of push service instances.

5. The system of claim 1, wherein the instructions are further executable by the one or more processors to:

identify any errors resulting from attempts to complete the scheduled work item; and

if execution of the scheduled work item is not successful, reschedule an additional attempt to pull the requested data from the assigned data source, wherein the additional attempt is rescheduled at least in part based on any identified errors.

6. The system of claim 1, wherein the command to pull requested data specifies an information sink that has been configured to accept the requested data.

7. The system of claim 1, wherein the push service is identified by performing an address resolution based on an address of an information sink.

8. The system of claim 1, wherein the instructions are further executable by the one or more processors to:

transform the requested data according to a protocol that is supported by the push service.

9. A method for pulling data from a plurality of data sources using a pull service, the method comprising:

retrieving information identifying an assigned data source of the plurality of data sources, wherein the pull service is tasked with pulling data from the assigned data source;

receiving a command to pull requested data;

scheduling an attempt to pull the requested data from the assigned data source, wherein the schedule is maintained at least in part as a queue of work items, wherein work items specify the requested data to be pulled from the assigned data source;

executing a scheduled work item; and

if the execution of the work item is successful, forwarding the requested data to a push service.

10. The method of claim 9, the method further comprising:

determining whether the requested data is maintained in a data source that is assigned to the pull service;

if the data source of the requested data is not assigned to the pull service;

identifying a capable pull service that is assigned to the data source; and

forwarding the command to the capable pull service.

11. The method of claim 10, wherein the pull service assigned to the data source of the requested data is determined by performing an address resolution based on an address of the data source of requested data.

12. The method of claim 9, the method further comprising:

accepting an additional assignment for pulling data from an additional assigned data source of the plurality of data sources, wherein the push service is tasked with pushing incoming data to an additional assigned data sink, and wherein the additional assignment results from adjustments to a partitioning of a plurality of data sinks among a plurality of push service instances.

13. The method of claim 9, the method further comprising:

identifying any errors resulting from execution of the scheduled work item; and

if an attempt to complete a work item is not successful, rescheduling an additional attempt to pull the requested data from the assigned data source, wherein the additional attempt is rescheduled at least in part based on any identified errors.

14. The method of claim 9, wherein the command to pull requested data specifies an information sink that has been configured to accept the requested data.

15. The method of claim 9, the method further comprising:

transforming the requested data according to a protocol that is supported by the push service.

16. A method, comprising:

receiving, at a pull agent service, information about a protocol that is defined for an information sink, the protocol being published by a push agent service;

receiving data using the pull agent service that pulls data from an endpoint source, wherein the pull agent service is a collection of pull endpoint hosting services that pull data from external information sources and each external information source is assigned to a particular pull endpoint hosting service from among the collection of pull endpoint hosting services;

transforming the data, by the pull agent service, into the protocol that is defined for the information sink, thereby creating first transformed data;

sending the first transformed data from the pull agent service to the push agent service;

performing additional transformations with respect to the first transformed data at the push agent service, thereby creating second transformed data; and

sending the second transformed data as messages to the information sink.

17. The method of claim 16, wherein the push agent service is a collection of push endpoint hosting services that push data to external information sinks.

18. The method of claim 17, wherein the collection of push endpoint hosting services partition the external information sinks.