Bit level comparator systems and methods

• US 9,507,814 B2
• Filed: 12/10/2013
• Issued: 11/29/2016
• Est. Priority Date: 12/10/2013
• Status: Active Grant

• Current Assignee: Vertafore Incorporated
• Original Assignee: Vertafore Incorporated
• Inventors: Alvarez, Raul
• Primary Examiner(s): KUDDUS, DANIEL A
• Application Number:
  US14/101,905
• Publication Number:
  US 20150161121A1
• Time in Patent Office: 1,085 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/2255 Hash tables

Abstract

A bit level file comparison system compares two file systems, each of which includes any number of individual files, to determine whether the file systems are identical at a bit level. A hashing function is applied to each file in the first file system to generate a hash value that is logically associated with the respective file in the first file system. The hashing function is applied to each file in the second file system to generate a hash value that is logically associated with the respective file in the second file system. The hash value associated with a file in the first file system is compared with the hash value associated with the corresponding file in the second file system to determine whether bit level differences between the respective file in the first file system and the second file system.

Claims

1. A method to compare a first file system including a number of machine readable digital files having a 1:

1 correspondence with a number of machine readable digital files included in each of a number of second file systems to detect bit-level differences between each of the number of machine readable digital files in the first file system and the corresponding machine readable digital file in some or all of the number of second file systems, the method comprising;

creating by at least one processor a catalog definition file that includes a number of file identifiers, each file identifier corresponding to and associated with one of a number of machine-readable digital files included in the first file system;

generating by the at least one processor via the catalog definition file a first catalog file containing a number of hash values, each of the number of hash values logically associated with a respective one of the number of file identifiers included in the catalog definition file;

storing data indicative of the first catalog file in at least one communicably coupled nontransitory storage medium by the at least one processor;

generating via the catalog definition file a respective second catalog file for each of the number of second file systems, each second catalog file including a number of hash values, each of the hash values logically associated with a respective one of the file identifiers included in the respective second catalog file;

storing each of the second catalog files in the at least one communicably coupled nontransitory storage medium by the at least one processor;

determining a symmetric difference between at least a portion of the data included in the first catalog file and the corresponding portion of the data included in the second catalog file;

determining whether the symmetric difference is a non-zero value;

responsive to determining that the symmetric difference is a non-zero value, generating an alert output; and

determining by the at least one processor whether each machine-readable digital file included in the first file system is identical at a bit level with a corresponding machine -readable digital file included in each of the number of second file systems by comparing on a file-by-file basis, the hash value associated with each of the number of file identifiers included in the stored first catalog file with the hash value logically associated with the corresponding one of the number of file identifiers included in each of the number of stored second catalog files.

2. The method of claim 1, further comprising:

inserting catalog header information into the catalog definition file.

3. The method of claim 1 wherein generating by the at least one processor via the catalog definition file, a first catalog file containing a number of hash values includes:

generating a first catalog file containing a number of hash values determined using an SHA1 hashing function; and

wherein generating by the at least one processor via the catalog definition file a respective second catalog file for each of the number of second file systems, each second catalog file containing a number of hash values includes;

generating a number of second catalog files, each of the number of second catalog files containing a number of hash values determined using an SHA1 hashing function.

4. The method of claim 1, further comprising:

for each instance where the hash value logically associated with one of the file identifiers included in the first catalog file does not match the hash value logically associated with the corresponding file identifier in some or all of the number of second catalog files, generating by the at least one processor an alert that includes data indicative of the identity of each file identifier for which the hash values do not match.

5. The method of claim 4 wherein generating an alert that includes data indicative of the identity of each file identifier for which the hash values do not match, further comprises:

generating by the at least one processor an alert that includes data indicative of;

the respective file identifier, the hash value logically associated with the respective file identifier included in the first catalog file and the hash value logically associated with the respective file identifier included in the second catalog file.

6. The method of claim 4 further comprising:

for each instance where the hash value logically associated with a file identifier included in the first catalog file does not match the hash value logically associated with the corresponding file identifier in some or all of the number of second catalog files, generating by the at least one processor an alert that includes data indicative of the identity of the one or more second catalog files including the respective non-matching hash value.

7. The method of claim 6 wherein generating an alert that includes data indicative of the identity of each of the one or more second catalog files for which the hash values do not match, further comprises:

generating by the at least one processor an alert that includes data indicative of;

the machine-readable digital file name associated with the respective file identifier and the hash value logically associated with the respective file identifier included in the first catalog file; and

, the machine-readable digital file name associated with the respective file identifier and the hash value logically associated with the respective file identifier in the respective second directory.

8. A nontransitory, computer readable media containing machine readable, processor executable, instructions that when executed by at least one processor, cause the at least one processor to function as a bit-level file comparator by:

creating a catalog definition file that includes a number of file identifiers, each file identifier corresponding to and associated with a machine-readable digital file included in a first file system;

generating via the catalog definition file a first catalog file containing a number of hash values, each of the number of hash values logically associated with a respective one of the number of file identifiers included in the catalog definition file;

storing the first catalog in at least one communicably coupled, nontransitory storage medium;

generating via the catalog definition file a respective second catalog file for each of a number of second file systems, each second catalog file including a number of hash values, each of the hash values logically associated with a respective one of a number of file identifiers included in the respective second catalog file;

storing each of the number of second catalogs in the at least one communicably coupled, nontransitory storage medium;

determining a symmetric difference between at least a portion of the data included in the first catalog file and the corresponding portion of the data included in the second catalog file;

determining whether the symmetric difference is a non-zero value;

responsive to determining that the symmetric difference is a non-zero value, generating an alert output; and

determining whether each machine-readable digital file included in the first file system is identical at a bit level with a corresponding machine-readable digital file included in each of the number of second file systems by comparing on a file-by-file basis, the hash value logically associated with each of the number of file identifiers included in the stored first catalog file with the hash value logically associated with the corresponding one of the number of file identifiers included in each of the number of stored second catalog files.

9. The nontransitory, computer readable media of claim 8, further comprising machine readable, processor executable, instructions that when executed by at least one processor, further cause the at least one processor to function as a bit-level file comparator by:

inserting catalog header information into the catalog definition file.

10. The nontransitory, computer readable media of claim 8 wherein generating via the catalog definition file a first catalog file containing a number of hash values, includes:

generating a first catalog file containing a number of hash values determined using an SHA1 hashing function; and

wherein generating via the catalog definition file a respective second catalog file for each of the number of second file systems, each second catalog file containing a number of hash values, includes;

generating a number of second catalog files, each of the number of second catalog files containing a number of hash values determined using an SHA1 hashing function.

11. The nontransitory, computer readable media of claim 8, further comprising machine readable, processor executable, instructions that cause the at least one processor to function as a bit-level file comparator by:

for each instance where the hash value logically associated with one of the file identifiers included in the first catalog file does not match the hash value logically associated with the corresponding file identifier in some or all of the number of second catalog files, generating an alert that includes data indicative of the identity of each file identifier for which the hash values do not match.

12. The nontransitory, computer readable media of claim 11 wherein the machine readable, processor executable, instructions that cause the at least one processor to function as a bit-level file comparator by generating an alert that includes data indicative of the identity of each file identifier for which the hash values do not match, further cause the at least one processor to function as a bit-level file comparator by:

generating an alert that includes data indicative of;

the respective file identifier, the hash value logically associated with the respective file identifier included in the first catalog file and the hash value logically associated with the respective file identifier included in the second catalog file.

13. The nontransitory, computer readable media of claim 11, further comprising machine readable, processor executable, instructions that cause the at least one processor to function as a bit-level file comparator by:

for each instance where the hash value logically associated with a file identifier included in the first catalog file does not match the hash value logically associated with the corresponding file identifier in some or all of the number of second catalog files, generating an alert that includes data indicative of the identity of the one or more second catalog files including the respective non-matching hash value.

14. The nontransitory, computer readable media of claim 13 wherein the machine readable, processor executable, instructions that cause the at least one processor to function as a bit-level file comparator by generating an alert that includes data indicative of the identity of the one or more second catalog files for which the logically associated hash values do not match, further cause the at least one processor to function as a bit-level file comparator by:

generating an alert that includes data indicative of;

the machine-readable digital file name associated with the respective file identifier and the hash value logically associated with the respective file identifier included in the first catalog file; and

, the machine-readable digital file name associated with the respective file identifier and the hash value logically associated with the respective file identifier in the respective second catalog file.

15. A file comparator system to compare a first file system including a number of machine readable digital files having a 1:

1 correspondence with a number of machine readable digital files included in each of a number of second file systems to detect bit-level differences between each of the number of machine readable digital files in the first file system and the respective machine readable digital file in each of the number of second file systems, the system comprising;

at least one nontransitory storage medium that, when in operation, stores data indicative of the number of machine readable digital files included in the first file system and data indicative of the number of machine readable digital files included in each of the number of second file systems;

at least one processor communicably coupled to the at least one nontransitory storage medium; and

at least one set of machine readable, processor executable, instructions included in the at least one nontransitory storage medium, that when executed by the at least one processor cause the at least one processor to;

create a catalog definition file that includes a number of file identifiers, each file identifier corresponding to and associated with one of a number of machine readable digital files included in the first file system;

generate via the catalog definition file a first catalog file containing a number of hash values, each of the number of hash values logically associated with a respective one of the number of file identifiers included in the catalog definition file;

store the first catalog in the at least one nontransitory storage medium;

generate via the catalog definition file a respective second catalog file for each of the number of second file systems, each second catalog file including a number of hash values, each of the hash values logically associated with a respective one of the number of file identifiers included in the respective second catalog file;

store each of the number of second catalog files in the at least one nontransitory storage medium;

determine a symmetric difference between at least a portion of the data included in the first catalog file and the corresponding portion of the data included in the second catalog file;

determine whether the symmetric difference is a non-zero value;

responsive to a determination that the symmetric difference is a non-zero value, generate an alert output; and

determine whether each machine-readable digital file included in the first file system is identical at a bit level with a corresponding machine-readable digital file included in each of the number of second file systems by comparing on a file-by-file basis, the hash value associated with each of the number of file identifiers included in the stored first catalog file with the hash value logically associated with the corresponding one of the number of file identifiers included in each of the number of stored second catalog files.

16. The file comparator system of claim 15 wherein the one or more sets of machine readable, processor executable instructions cause the at least one processor to further:

insert catalog header information into the catalog definition file.

17. The file comparator system of claim 15 wherein the one or more sets of machine readable, processor executable instructions that cause the at least one processor to generate via the catalog definition file a first catalog file containing a number of hash values, cause the at least one processor to further:

generate a first catalog file containing a number of hash values determined using an SHA1 hashing function; and

wherein the one or more sets of machine readable, processor executable instructions that cause the at least one processor to generate via the catalog definition file, a respective second catalog file for each of the number of second file systems, each second catalog file including a number of hash values further cause the at least one processor to;

generate a number of second catalog files, each of the number of second catalog files containing a number of hash values determined using an SHA1 hashing function.

18. The file comparator system of claim 15 wherein the one or more sets of machine readable, processor executable instructions cause the at least one processor to further:

for each instance where the hash value logically associated with one of the file identifiers included in the first catalog file does not match the hash value logically associated with the corresponding file identifier in some or all of the number of second catalog files, generate an alert that includes data indicative of the identity of each file identifier for which the hash values do not match.

19. The file comparator system of claim 18 wherein the one or more sets of machine readable, processor executable instructions that cause the at least one processor to generate an alert that includes data indicative of the identity of each file identifier for which the hash values do not match, cause the at least one processor to further:

generate an alert that includes data indicative of;

the respective file identifier, the hash value logically associated with the respective file identifier included in the first catalog file, and the hash value logically associated with the respective file identifier included in the second catalog file.

20. The file comparator system of claim 18 wherein the one or more sets of machine readable, processor executable instructions further cause the at least one processor to:

for each instance where the hash value logically associated with the file identifier included in the first catalog file does not match the hash value logically associated with the corresponding file identifier in some or all of the number of second catalog files, generate an alert that includes data indicative of the identity of the one or more second catalog files including the respective non-matching hash value.

21. The file comparator system of claim 20 wherein the one or more sets of machine readable, processor executable instructions that cause the at least one processor to generate an alert that includes data indicative of the identity of the one or more second catalog files for which the hash values do not match, cause the at least one processor to further:

generate an alert that includes data indicative of;

the machine-readable digital file name associated with the respective file identifier and the hash value logically associated with the respective file identifier included in the first catalog file; and

, the machine-readable digital file name associated with the respective file identifier and the hash value logically associated with the respective file identifier in the respective second catalog file.