Data collecting method, data collecting apparatus and network management device

• US 8,954,971 B2
• Filed: 07/22/2010
• Issued: 02/10/2015
• Est. Priority Date: 12/30/2009
• Status: Active Grant

• Current Assignee: ZTE Corporation
• Original Assignee: ZTE Corporation
• Inventors: Zhao, Rui
• Primary Examiner(s): ZHE, MENG YAO
• Application Number:
  US13/519,192
• Publication Number:
  US 20120297393A1
• Time in Patent Office: 1,664 Days
• Field of Search: None
• US Class Current: 718/102
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system

Abstract

The present invention provides a data collection method and apparatus and a network management device. The method includes: a network management device collecting data files to be processed reported by a network element device; assigning the data files to be processed as a plurality of tasks; adding the assigned tasks into a task queue and extracting tasks from the task queue one by one for processing. According to the present invention, the task work load can be automatically adjusted according to the computer configuration and parameter configuration, and the maximum efficiency of data processing can be achieved under different scenarios.

Claims

1. A data collection method, comprising:

a network management device collecting data files to be processed reported by a network element device;

assigning the data files to be processed as a plurality of tasks; and

adding the plurality of tasks into a task queue and extracting tasks from the task queue one by one for processing;

wherein the step of assigning the data files to be processed as a plurality of tasks comprises;

evaluating a total quantity or capacity of the data files to be processed;

evaluating local calculation ability;

obtaining local configuration parameters;

calculating task payload of each thread according to the quantity or capacity of the data files to be processed, the local calculation ability and the configuration parameter; and

assigning the data files to be processed as the plurality of tasks based on the task payload;

wherein the local calculation ability comprises;

local CPU processing speed and memory capacity, and the configuration parameter comprises;

a number of threads and maximum activation time of the threads;

wherein the task payload of each thread is equal to the total data quantity or capacity divided by a number of processing batch and then divided by the number of threads;

wherein the number of processing batch=Max (the total data CPU processing time/(the number of threads*the maximum activation time), the total capacity of the data/memory capacity), wherein, Max represents to take a maximum;

wherein the total data CPU processing time is equal to the total quantity or total capacity of the data files to be processed divided by the CPU processing speed.

3. The method as claimed in claim 1, wherein

contents of each task are parsing data files under the leaf directories covered by each task.

2. A network management device, comprising a processor, and a computer storage media storing instruction that, when executed, causes the device to:

obtain data files to be processed reported by a network element device;

assign the data files to be processed into a plurality of tasks;

add the plurality of tasks into a task queue, and extract tasks from the task queue one by one for processing;

wherein the instructions cause the device to assign the data files to be processed into the plurality of tasks by way of the following manner;

evaluating a total quantity or capacity of the data files to be processed;

evaluating local calculation ability;

obtaining local configuration parameters;

calculating task payload of each thread according to the quantity or capacity of the data files to be processed, the local calculation ability and the configuration parameter; and

assigning the data files to be processed as the plurality of tasks based on the task payload;

wherein the local calculation ability comprises;

local CPU processing speed and memory capacity, and the configuration parameter comprises;

a number of threads and maximum activation time of the threads;

wherein the task payload of each thread is equal to the total data quantity or capacity divided by a number of processing batch and then divided by the number of threads;

wherein the number of processing batch=Max (the total data CPU processing time/(the number of threads*the maximum activation time), the total capacity of the data/memory capacity), wherein, Max represents to take a maximum;

wherein the total data CPU processing time is equal to the total quantity or total capacity of the data files to be processed divided by the CPU processing speed.

4. The device as claimed in claim 2, wherein

contents of each task are parsing data files under the leaf directories covered by each task.