Intelligent, scalable, low-overhead mechanism for data retrieval in a distributed network environment

• US 9,418,110 B1
• Filed: 06/30/2008
• Issued: 08/16/2016
• Est. Priority Date: 06/30/2008
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Shmuylovich, Samuil, Ninan, Anoop George, Farizon, Boris
• Primary Examiner(s): Syed, Farhan
• Application Number:
  US12/164,595
• Time in Patent Office: 2,969 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24565 Triggers; Constraints
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70

Abstract

Techniques disclosed herein describe a data retrieval process for storing management data from a network environment using an efficient polling-based approach without sacrificing the real-time aspect of data retrieval that a push model provides. The process is highly scalable and reliable, and is useful in Network Address Translation (NAT) environments because the number of holes in a NAT firewall is minimized. The data retrieval component uses intelligent polling to retrieve data and store the data in a central database without overloading the system. A data retrieval component polls a data collection agent for a dataset. The data collection agent transfers the data set and a poll indication of when a subsequent data set will be ready for transfer. Subsequent polling is based on a processor load at a data retrieval component.

Claims

1. A computer-implemented method for data retrieval by a data collection agent operating on a host computer system in a distributed network, the method comprising:

collecting data from at least one managed resource in the distributed network into a dataset at the data collection agent according to a predefined policy including a data collection frequency;

receiving a poll from a data retrieval component in the distributed network to retrieve the dataset from the data collection agent, wherein the poll causes the data collection agent to transfer the dataset to the data retrieval component according to a firewall hole dedicated to the data retrieval component and wherein the predefined policy further includes a threshold for receiving data retrieval polls from the data retrieval component;

responsive to the poll, transferring the dataset to the data retrieval component, and transferring a poll indication to the data retrieval component indicating when a subsequent dataset will be available for retrieval from the data collection agent by the data retrieval component according to a subsequent poll from the data retrieval component;

collecting data from the at least one managed resource in the distributed network into a subsequent dataset at the data collection agent according to the predefined policy; and

receiving the subsequent poll from the data retrieval component in the distributed network to retrieve the subsequent dataset from the data collection agent at a time based on the poll indication transferred from the data collection agent to the data retrieval component indicating when the subsequent dataset would be available for retrieval from the data collection agent by the data retrieval component.

2. The method of claim 1, wherein receiving a subsequent poll, from the data retrieval component, to retrieve the subsequent dataset is further based on a current processor load at the data retrieval component.

3. The method of claim 2, further comprising responsive to the subsequent poll, transferring the subsequent dataset.

4. The method of claim 3, further comprising transferring multiple datasets collected between an initial poll and a subsequent poll.

5. The method of claim 4, wherein polls received are sent from within a network address translation environment.

6. The method of claim 1, wherein responsive to data retrieval activity that violates policy, executing a corrective action.

7. The method of claim 6, wherein executing a corrective action comprises notifying a management server of the policy violation.

8. The method of claim 7, wherein the policy violation of the data retrieval activity is an absence of polling requests within a predetermined period of time.

9. A computer-implemented method for data retrieval by a data retrieval component computerized device in a distributed network, the method comprising:

polling a data collection agent in the distributed network to retrieve a dataset, containing data collected from at least one managed resource in the distributed network by the data collection agent according to a predefined policy including a data collection frequency and a threshold for receiving data retrieval polls from the data retrieval component, wherein the poll causes the data collection agent to transfer the dataset to the data retrieval component according to a firewall hole dedicated to the data retrieval component;

receiving the dataset from the data collection agent, and receiving a poll indication from the data collection agent indicating when a subsequent dataset will be available for retrieval from the data collection agent by the data retrieval component according to a subsequent poll to the data collection agent; and

subsequently polling the data collection agent in the distributed network to retrieve the subsequent dataset from the data collection agent, containing data collected from at least one managed resource in the distributed network by the data collection agent according to the predefined policy, at a time based on the poll indication transferred from the data collection agent to the data retrieval component indicating when the subsequent dataset would be available for retrieval from the data collection agent by the data retrieval component.

10. The method of claim 9, wherein polling the data collection agent for the subsequent dataset, is based on a current processor load at the data retrieval component.

11. The method of claim 10, further comprising refraining from polling subsequent datasets when the current processor load exceeds a predefined level of activity.

12. The method of claim 11, further comprising notifying a management server when polling activity fails to meet a predefined retrieval policy.

13. The method of claim 10, wherein the subsequent polling retrieves all datasets collected since a previous poll.

14. The method of claim 13, further comprising storing retrieved data in a central database.

15. The method of claim 14, wherein polling the data collection agent comprises contacting a data collection agent assigned by a management server, and presenting a ticket received from the management server.

16. A system comprising:

a data collection agent operating on a host computer system configured to collect data within a distributed network into datasets containing data collected from at least one managed resource in the distributed network by the data collection agent according to a predefined policy including a data collection frequency;

a data retrieval component computerized device in communication with the data collection agent configured to poll the data collection agent in the distributed network to retrieve the datasets, wherein the predefined policy further includes a threshold for receiving data polls from the data retrieval component;

wherein the data collection agent includes computer program code to execute a process that collects data from at least one managed resource in the distributed network into a dataset at the data collection agent according to the policy;

wherein the data retrieval component includes computer program code to execute a process that polls the data collection agent in the distributed network to retrieve the dataset from the data collection agent, wherein the poll causes the data collection agent to transfer the dataset to the data retrieval component according to a firewall hole dedicated to the data retrieval component;

wherein the data collection agent includes computer program code to execute a process that receives the poll from the data retrieval component in the distributed network to retrieve the dataset from the data collection agent and, responsive to the poll, transfers the dataset from the data collection agent to the data retrieval component, and transfers a poll indication from the data collection agent to the data retrieval component indicating when a subsequent dataset will be available for retrieval from the data collection agent by the data retrieval component according to a subsequent poll from the data retrieval component;

wherein the data retrieval component includes computer program code to execute a process that receives the dataset from the data collection agent, and receives the poll indication from the data collection agent indicating when the subsequent dataset will be available for retrieval from the data collection agent by the data retrieval component according to the subsequent poll to the data collection agent;

wherein the data collection agent includes computer program code to execute a process that collects data from the at least one managed resource in the distributed network into the subsequent dataset at the data collection agent according to the predefined policy;

wherein the data retrieval component includes computer program code to execute a process that subsequently polls the data collection agent in the distributed network to retrieve the subsequent dataset from the data collection agent at a time based on the poll indication transmitted from the data collection agent to the data retrieval component indicating when the subsequent dataset would be available for retrieval from the data collection agent by the data retrieval component; and

wherein the data collection agent includes computer program code to execute a process that receives the subsequent poll from the data retrieval component in the distributed network to retrieve the subsequent dataset from the data collection agent at the time based on the poll indication transferred from the data collection agent to the data retrieval component indicating when the subsequent dataset would be available for retrieval from the data collection agent by the data retrieval component.

17. The system of claim 16, wherein the management server, responsive to user configuration, defines a data retrieval component assignment type as one of static assignment and dynamic assignment, and wherein the management server defines a connection type, between the data retrieval component and the data collection agent, is one of persistent or transient, and wherein the management server defines a protocol for retrieving data.

18. The system of claim 17, wherein data collected by the data collection agent includes performance data.

19. The system is claim 18, wherein the data collection agent has further computer program code to execute a process that notifies the management server when polling by the data retrieval component violates a data retrieval process.