Analyzing binary data streams to identify embedded record structures

• US 8,316,034 B2
• Filed: 10/20/2010
• Issued: 11/20/2012
• Est. Priority Date: 10/20/2010
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Guirard, Andre P.
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US12/908,018
• Publication Number:
  US 20120102056A1
• Time in Patent Office: 762 Days
• Field of Search: 707/2, 707/100, 707/755, 370/412
• US Class Current: 707/755
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

According to one embodiment of the present invention, a system identifies and extracts data structures embedded within a data stream, and comprises a computer system including at least one processor. The system analyzes the data stream and identifies data structure definitions corresponding to analyzed data stream portions. The data structure definitions each define a structure of a corresponding data structure, and are configurable to accommodate various types of data structures within the data stream. The corresponding data structures within the data stream are generated by extracting information from the data stream in accordance with the data structure definitions. Embodiments of the present invention further include a method and computer program product for identifying and extracting data structures embedded within a data stream in substantially the same manner described above.

Claims

1. A computer-implemented method of identifying and extracting data structures embedded within a data stream of binary data comprising:

analyzing the data stream of binary data including a data file with a plurality of different types of data structures and identifying data structure definitions corresponding to analyzed data stream portions, wherein the data structure definitions each define a structure and arrangement of a corresponding data structure, map fields of the corresponding data structure to binary data, and are configurable to accommodate various types of data structures within the data stream; and

generating the corresponding data structures of the data file from the binary data within the data stream by extracting information from the data stream in accordance with the identified data structure definitions.

2. The computer-implemented method of claim 1, wherein at least one data structure definition indicates one or more other data structures expected subsequently within the data stream, and the method further includes:

generating a representation of a sequence of the expected data structures within the data stream by storing indicators of the expected data structures in corresponding sections of a storage unit, wherein the storage unit sections are each associated with a corresponding location within the sequence;

wherein the identifying data structure definitions includes comparing the analyzed data stream portions to data structure definitions corresponding to the expected data structures within the storage unit.

3. The computer-implemented method of claim 1, wherein the data structures embedded in the data stream include records.

4. The computer-implemented method of claim 1, further including:

examining the generated data structures to retrieve certain types of data contained in the data structures.

5. The computer-implemented method of claim 1, wherein the data stream includes a plurality of data files and the data structures for each file are generated, and wherein the method further includes:

comparing the generated data structures for each file to determine differences between the files; and

reporting the differences between the files using descriptors for the data structures discernable by a user, wherein the data structure definitions corresponding to the generated data structures include the descriptors.

6. The computer-implemented method of claim 1, wherein the data stream includes a first set of data structures and a second set of data structures and the data structures for each set are generated, and wherein the method further includes:

comparing the generated data structures for each set to determine differences between the sets that affect operation of a system.

7. The computer-implemented method of claim 6, wherein the data structure definitions for the first and second sets of data structures indicate portions of the data structures for use in the comparison to determine the differences that affect operation of the system.

8. The computer-implemented method of claim 6, wherein the comparing the first and second sets of data structures includes:

generating copies of data within ones of the first set of data structures being of the same type but not identical to data structures within the second set; and

manipulating the data within the data copies and comparing the manipulated copies to the second set of data structures to determine the differences.

9. A system for identifying and extracting data structures embedded within a data stream of binary data comprising:

a computer system including at least one processor configured to;

analyze the data stream of binary data including a data file with a plurality of different types of data structures and identify data structure definitions corresponding to analyzed data stream portions, wherein the data structure definitions each define a structure and arrangement of a corresponding data structure, map fields of the corresponding data structure to binary data, and are configurable to accommodate various types of data structures within the data stream; and

generate the corresponding data structures of the data file from the binary data within the data stream by extracting information from the data stream in accordance with the identified data structure definitions.

10. The system of claim 9, wherein at least one data structure definition indicates one or more other data structures expected subsequently within the data stream, and the at least one processor is further configured to:

generate a representation of a sequence of the expected data structures within the data stream by storing indicators of the expected data structures in corresponding sections of a storage unit, wherein the storage unit sections are each associated with a corresponding location within the sequence;

wherein the identifying data structure definitions includes comparing the analyzed data stream portions to data structure definitions corresponding to the expected data structures within the storage unit.

11. The system of claim 9, wherein the data structures embedded in the data stream include records.

12. The system of claim 9, wherein the at least one processor is further configured to:

examine the generated data structures to retrieve certain types of data contained in the data structures.

13. The system of claim 9, wherein the data stream includes a plurality of data files and the data structures for each file are generated, and wherein the at least one processor is further configured to:

compare the generated data structures for each file to determine differences between the files; and

report the differences between the files using descriptors for the data structures discernable by a user, wherein the data structure definitions corresponding to the generated data structures include the descriptors.

14. The system of claim 9, wherein the data stream includes a first set of data structures and a second set of data structures and the data structures for each set are generated, and wherein the at least one processor is further configured to:

compare the generated data structures for each set to determine differences between the sets that affect operation of a system.

15. The system of claim 14, wherein the data structure definitions for the first and second sets of data structures indicate portions of the data structures for use in the comparison to determine the differences that affect operation of the system.

16. The system of claim 14, wherein the comparing the first and second sets of data structures includes:

generating copies of data within ones of the first set of data structures being of the same type but not identical to data structures within the second set; and

manipulating the data within the data copies and comparing the manipulated copies to the second set of data structures to determine the differences.

17. A computer program product for identifying and extracting data structures embedded within a data stream of binary data comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising computer readable program code configured to;

analyze the data stream of binary data including a data file with a plurality of different types of data structures and identify data structure definitions corresponding to analyzed data stream portions, wherein the data structure definitions each define a structure and arrangement of a corresponding data structure, map fields of the corresponding data structure to binary data, and are configurable to accommodate various types of data structures within the data stream; and

generate the corresponding data structures of the data file from the binary data within the data stream by extracting information from the data stream in accordance with the identified data structure definitions.

18. The computer program product of claim 17, wherein at least one data structure definition indicates one or more other data structures expected subsequently within the data stream, and the computer readable program code further includes computer readable program code configured to:

generate a representation of a sequence of the expected data structures within the data stream by storing indicators of the expected data structures in corresponding sections of a storage unit, wherein the storage unit sections are each associated with a corresponding location within the sequence;

wherein the identifying data structure definitions includes comparing the analyzed data stream portions to data structure definitions corresponding to the expected data structures within the storage unit.

19. The computer program product of claim 17, wherein the data structures embedded in the data stream include records.

20. The computer program product of claim 17, wherein the computer readable program code further includes computer readable program code configured to:

examine the generated data structures to retrieve certain types of data contained in the data structures.

21. The computer program product of claim 17, wherein the data stream includes a plurality of data files and the data structures for each file are generated, and wherein the computer readable program code further includes computer readable program code configured to:

compare the generated data structures for each file to determine differences between the files; and

report the differences between the files using descriptors for the data structures discernable by a user, wherein the data structure definitions corresponding to the generated data structures include the descriptors.

22. The computer program product of claim 17, wherein the data stream includes a first set of data structures and a second set of data structures and the data structures for each set are generated, and wherein the computer readable program code further includes computer readable program code configured to:

compare the generated data structures for each set to determine differences between the sets that affect operation of a system.

23. The computer program product of claim 22, wherein the data structure definitions for the first and second sets of data structures indicate portions of the data structures for use in the comparison to determine the differences that affect operation of the system.

24. The computer program product of claim 22, wherein the comparing the first and second sets of data structures includes:

generating copies of data within ones of the first set of data structures being of the same type but not identical to data structures within the second set; and

manipulating the data within the data copies and comparing the manipulated copies to the second set of data structures to determine the differences.