Systems, methods, and computer program products to ingest, process, and output large data

• US 9,582,541 B2
• Filed: 12/20/2013
• Issued: 02/28/2017
• Est. Priority Date: 02/01/2013
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Sareen, Apoorva, Shriyan, Rajesh, Singh, Vikram Kumar, Hambleton, Jordan Patrick, Sankaran, Senthil Kumar Laguduwa
• Primary Examiner(s): Arjomandi, Noosha
• Application Number:
  US14/137,707
• Publication Number:
  US 20140222843A1
• Time in Patent Office: 1,166 Days
• Field of Search: 707/755
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method for ingesting data, the method comprising receiving a file including a plurality of portions, each of the portions corresponding to a respective characteristic of a hardware or software component in a deployed system; breaking the file into a plurality of sections, each of the sections corresponding to a respective portion of the file; for each section, adding first metadata to uniquely identify the file and provide a name for the section; and distributing the sections to a plurality of container files.

Claims

1. A method, comprising:

receiving a file comprising a header, the file including a plurality of portions, each of the portions corresponding to a respective characteristic of a hardware or software component in a deployed system;

classifying the file as actionable in response to examining the header and discerning therefrom that the file corresponds to an error on a deployed system;

classifying the file as non-actionable in response to examining the header and discerning therefrom that the file corresponds to a periodic report from a deployed system;

breaking the file into a plurality of sections, each of the sections corresponding to a respective portion of the file;

for each section, adding first metadata to uniquely identify the file and provide a name for the section; and

distributing the sections to a plurality of container files.

2. The method of claim 1, the container files comprising sequence files.

3. The method of claim 1, the file comprising a packet sent over the Internet to a remote customer support site.

4. The method of claim 1, comprising:

examining the header by a rules engine to classify the file as actionable or non-actionable.

5. The method of claim 1, comprising:

sending a message to a customer case generator in response to classifying the file as actionable; and

generating a customer case to investigate an error indicated by the file.

6. The method of claim 1, comprising:

discerning that a first one of the sections has a data size that exceeds a threshold; and

breaking the first one of the sections into multiple sub-sections and adding second metadata to indicate a relationship among the sub-sections, the second metadata including a flag to indicate that a sub-section is less than a complete section and a sequential counter to indicate a sub-section number within the first one of the sections, each of the sub-sections including the first metadata; and

distributing the sub-sections to the plurality of container files.

7. The method of claim 1, the first metadata to uniquely identify the file comprising an identification of the deployed system and a time that the file was generated.

8. The method of claim 1 in which each section is saved to its respective container file as a key value pair, wherein a key includes a unique identifier of the file and the name for the section, and wherein a value includes raw data content from the section.

9. A system comprising:

a storage system; and

an ingestion server in communication with the storage system, the ingestion server executing code to;

receive a system information file comprising a header over a network from a remote, deployed system, the system information file including a plurality of portions, wherein each portion describes a respective characteristic of a hardware or software component of the deployed system;

classify the system information file as actionable in response to examining the header and discerning therefrom that the system information file corresponds to an error on a deployed system;

classify the system information file as non-actionable in response to examining the header and discerning therefrom that the system information file corresponds to a periodic report from a deployed system;

divide the system information file into a plurality of sections, each of the sections corresponding to a respective portion of the system information file;

within each section, add a unique identifier of the system information file and a name of the section;

store the plurality of sections to a plurality of container files in the storage system, each of the sections being added as a key value pair, wherein a key includes the unique identifier of the system information file and a name of a respective section.

10. The system of claim 9, comprising a value of a key value pair comprising raw data from the respective section.

11. The system of claim 9, the unique identifier of the system information file comprising an identification of the deployed system and a time that the system information file was generated.

12. The system of claim 9, comprising:

a distributed computing system in communication with the ingestion server and operable to perform Map Reduce operations on data in the container files.

13. The system of claim 9, the ingestion server to:

discern that a first one of the sections has a data size that exceeds a threshold;

break the first one of the sections into multiple sub-sections and adding metadata to indicate a relationship among the sub-sections, the metadata including a flag to indicate that a sub-section is less than a complete section and a sequential counter to indicate a sub-section number within the first one of the sections; and

distribute the sub-sections to the plurality of container files.

14. A computer readable medium tangibly recording computer program logic comprising a plurality of instructions, that when executed, cause a system to:

parse a file comprising a header into a plurality of sections, wherein the file includes multiple portions, wherein each of the portions comprises information directed to a respective characteristic of a deployed system, further wherein each of the sections corresponds to a respective portion;

classify the file as actionable in response to examining the header and discerning therefrom that the file corresponds to an error on the deployed system;

classify the file as non-actionable in response to examining the header and discerning therefrom that the file corresponds to a periodic report from the deployed system; and

distribute the sections among a plurality of container files in a distributed storage system by storing each section as a key value pair, each respective key including a unique identifier of the file and a name of a respective section.

15. The computer readable medium of claim 14, a value of a key value pair includes raw data from the respective section.

16. The computer readable medium of claim 14, the unique identifier of the file comprising an identification of the deployed system and a time that the file was generated.

17. The computer readable medium of claim 14, comprising the plurality of instructions, that when executed, cause the system to:

discern that a first one of the sections has a data size that exceeds a threshold;

break the first one of the sections into multiple sub-sections and adding metadata to indicate a relationship among the sub-sections, the metadata including a flag to indicate that a sub-section is less than a complete section and a sequential counter to indicate a sub-section number within the first one of the sections; and

distribute the sub-sections to the plurality of container files.

18. The computer readable medium of claim 14, comprising the plurality of instructions, that when executed, cause the system to:

examine the header by a rules engine to classify the file as actionable or non-actionable.