Systems, Methods, and computer Program Products to Ingest, Process, and Output Large Data

• US 20140222843A1
• Filed: 12/20/2013
• Published: 08/07/2014
• Est. Priority Date: 02/01/2013
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Sareen, Apoorva, Shriyan, Rajesh, Singh, Vikram Kumar, Hambleton, Jordan Patrick, Sankaran, Senthil Kumar Laguduwa
• Granted Patent: US 9,582,541 B2
• Time in Patent Office: Days
• US Class Current: 707/755
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method for ingesting data, the method comprising receiving a file including a plurality of portions, each of the portions corresponding to a respective characteristic of a hardware or software component in a deployed system; breaking the file into a plurality of sections, each of the sections corresponding to a respective portion of the file; for each section, adding first metadata to uniquely identify the file and provide a name for the section; and distributing the sections to a plurality of container files.

Claims

1. A method for ingesting data, the method comprising:

receiving a file including a plurality of portions, each of the portions corresponding to a respective characteristic of a hardware or software component in a deployed system;

breaking the file into a plurality of sections, each of the sections corresponding to a respective portion of the file;

for each section, adding first metadata to uniquely identify the file and provide a name for the section; and

distributing the sections to a plurality of container files.

2. The method of claim 1 in which the container files comprise sequence files.

3. The method of claim 1 in which the file comprises a packet sent over the Internet to a remote customer support site.

4. The method of claim 1 in which the file comprises a header, the method further comprising:

examining the header by a rules engine to classify the file as actionable or non-actionable.

5. The method of claim 4 in which classifying the file comprises:

classifying the file as actionable in response to examining the header and discerning therefrom that the file corresponds to an error on a deployed system; and

classifying the file as non-actionable in response to examining the header and discerning therefrom that the file corresponds to a periodic report from a deployed system.

6. The method of claim 4 further comprising:

sending a message to a customer case generator in response to classifying the file as actionable; and

generating a customer case to investigate an error indicated by the file.

7. The method of claim 1 further comprising:

discerning that a first one of the sections has a data size that exceeds a threshold; and

breaking the first one of the sections into multiple sub-sections and adding second metadata to indicate a relationship among the sub-sections, the second metadata including a flag to indicate that a sub-section is less than a complete section and a sequential counter to indicate a sub-section number within the first one of the sections, each of the sub-sections including the first metadata; and

distributing the sub-sections to the plurality of container files.

8. The method of claim 1 in which the first metadata to uniquely identify the file comprises an identification of the deployed system and a time that the file was generated.

9. The method of claim 1 in which each section is saved to its respective container file as a key value pair, wherein a key includes a unique identifier of the file and the name for the section, and wherein a value includes raw data content from the section.

10. A system comprising:

a storage system; and

an ingestion server in communication with the storage system, the ingestion server executing code to provide the following functionality;

receiving a system information file over a network from a remote, deployed system, the system information file including a plurality of portions, wherein each portion describes a respective characteristic of a hardware or software component of the deployed system;

dividing the system information file into a plurality of sections, each of the sections corresponding to a respective portion of the system information file;

within each section, adding a unique identifier of the system information file and a name of the section;

storing the plurality of sections to a plurality of container files in the storage system, each of the sections being added as a key value pair, wherein a key includes the unique identifier of the system information file and a name of a respective section.

11. The system of claim 10 in which a value of a key value pair includes raw data from the respective section.

12. The system of claim 10 in which the unique identifier of the system information file comprises an identification of the deployed system and a time that the system information file was generated.

13. The system of claim 10, further comprising:

a distributed computing system in communication with the ingestion server and operable to perform Map Reduce operations on data in the container files.

14. The system of claim 10, in which the ingestion server further provides the following functionality:

discerning that a first one of the sections has a data size that exceeds a threshold; and

breaking the first one of the sections into multiple sub-sections and adding metadata to indicate a relationship among the sub-sections, the metadata including a flag to indicate that a sub-section is less than a complete section and a sequential counter to indicate a sub-section number within the first one of the sections; and

distributing the sub-sections to the plurality of container files.

15. A computer program product having a computer readable medium tangibly recording computer program logic for ingesting data, the computer program product comprising:

code to parse a file into a plurality of sections, wherein the file includes multiple portions, wherein each of the portions comprises information directed to a respective characteristic of a deployed system, further wherein each of the sections corresponds to a respective portion; and

code to distribute the sections among a plurality of container files in a distributed storage system by storing each section as a key value pair, each respective key including a unique identifier of the file and a name of a respective section.

16. The computer program product of claim 15 in which a value of a key value pair includes raw data from the respective section.

17. The computer program product of claim 15 in which the unique identifier of the file comprises an identification of the deployed system and a time that the file was generated.

18. The computer program product of claim 15, further comprising:

code to discern that a first one of the sections has a data size that exceeds a threshold; and

code to break the first one of the sections into multiple sub-sections and adding metadata to indicate a relationship among the sub-sections, the metadata including a flag to indicate that a sub-section is less than a complete section and a sequential counter to indicate a sub-section number within the first one of the sections

distributing the sub-sections to the plurality of container files.

19. The computer program product of claim 15 in which the file comprises a header, the computer program product further comprising:

code to examine the header by a rules engine to classify the file as actionable or non-actionable.

20. The computer program product of claim 15 which the code to examiner comprises:

code to classify the file as actionable in response to examining the header and discerning therefrom that the file corresponds to an error on the deployed system; and

code to classify the file as non-actionable in response to examining the header and discerning therefrom that the file corresponds to a periodic report from the deployed system.