IDENTIFICATION OF PORTIONS OF DATA

• US 20190324645A1
• Filed: 08/01/2017
• Published: 10/24/2019
• Est. Priority Date: 08/02/2016
• Status: Active Grant

• Current Assignee: Child Rescue Coalition, Inc.
• Original Assignee: Child Rescue Coalition, Inc.
• Inventors: GARCIA-FERNANDEZ, Jose Omar, WILTSE, William Scott
• Granted Patent: US 11,263,177 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/152 using file content signatures, e.g. hash values
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: H04L 9/0643 Hash functions, e.g. MD5, SHA, HMAC or f9 MAC

Abstract

Disclosed is a novel system and process for automating the process of identifying deleted file chunks. The present invention has two components. A client component to identify data chunks and a server component for storage and indexing technology for the over 1 billion records relating to the data chunks necessary to run the software.

Claims

1. A computer-implemented method for identifying portions of data on a storage media, the method comprising:

accessing a storage media commutatively coupled to a first computer;

determining if the storage media has a block size with a multiple of 4096 bytes;

in response to the storage media having a block size that is a multiple of 4096 bytes performing;

a) retrieving a next 4096 byte cluster and position;

b) using a first hash function on the 4096 byte cluster to produce a first hash value;

c) applying a first bloom filter to the first hash value;

d) in response to the first bloom filter returning a possibility of the first hash value in a first set of data, performing

e) using a second hash function on the 4096 byte cluster to produce a second hash value;

f) applying a second bloom filter to the second hash value;

g) in response to the second bloom filter returning a possibility of the second hash value in a second set of data, transmitting the second hash value and the position to a second computer;

h) determining whether there are more 4096 byte clusters to be examined; and

.

i) in response to more 4096 byte clusters to be examined, returning to step a.

2. The computer-implemented method of claim 1, wherein the second bloom filter returning the possibility of the second hash value in the second set of data, comparing the second hash value against a blocked list, and transmitting the second hash value and the position to the second computer in response to the second hash value not in the blocked list.

3. The computer-implemented method of claim 2, wherein in response to the second bloom filter returning a possibility of the second hash value in the second set of data, caching the second hash value and the position in a group until one or more of:

all the 4096 byte clusters are examined,

a period of time has expired, or

a threshold of a cache size has been reached,

and then transmitting the group to the second computer.

4. The computer-implemented method of claim 3, further comprising:

receiving results data of matching the group to a database of second hash values and physical block identifiers.

5. The computer-implemented method of claim 4, wherein the results data includes one or more of:

a number of matches,

an identification of the storage media,

a hash function of a full file that has been matched,

a percentage of blocks that have been matched to the full file, or

a mapping of logical blocks that have been matched to physical blocks on the storage media.

6. The computer-implemented method of claim 1, wherein the first hash function is a CRC64 function and the second hash function is a MD-160 hash function.

7. The computer-implemented method of claim 1, wherein the bytes are unallocated on the storage media or are allocated on the storage media.

8. The computer-implemented method of claim 1, further comprising:

j) in response to the storage media not having a block size that is a multiple of 4096 bytes, but is a multiple of 512 bytes performing;

k) retrieving a next 512 byte cluster and position;

1) using a third hash function on the 512 byte cluster to produce a third hash value;

m) applying a third bloom filter to the third hash value;

n) in response to the third bloom filter returning a possibility of the third hash value in a third set of data, performing

p) using a fourth hash function on the 512 byte cluster to produce a fourth hash value;

q) applying a fourth bloom filter to the fourth hash value;

r) in response to the fourth bloom filter returning a possibility of the fourth hash value in a fourth set of data, transmitting the fourth hash value and the position to the second computer; and

h) determining whether there are more 512 byte clusters to be examined; and

.

s) in response to more 512 byte clusters to be examined, returning to step k.

9. The computer-implemented method of claim 8, wherein the fourth bloom filter returning the possibility of the fourth hash value in the fourth set of data, comparing the fourth hash value against a blocked list, and transmitting the fourth hash value and the position to the second computer in response to the fourth hash value not in the blocked list.

10. The computer-implemented method of claim 9, wherein in response to the fourth bloom filter returning a possibility of the fourth hash value in a fourth set of data, caching the fourth hash value and the position in a group until one or more of:

all the 512 byte clusters are examined,

a period of time has expired, or

a threshold of a cache size has been reached,

and then transmitting the group to the second computer.

11. The computer-implemented method of claim 10, further comprising:

receiving results data of matching the group to a database of fourth hash values and physical blocks identifiers.

12. The computer-implemented method of claim 11, wherein the results data includes one or more of:

a number of matches,

an identification of the storage media,

a hash function of a full file that has been matched,

a percentage of blocks that have been matched to the full file, or

a mapping of logical blocks that have been matched to physical blocks on the storage media.

13. The computer-implemented method of claim 8, wherein the third hash function is a CRC64 function and the fourth hash function is a MD-160 hash function.

14. The computer-implemented method of claim 8, wherein the bytes are unallocated on the storage media or are allocated on the storage media.

15. A computer-implemented method for identifying portions of data on a storage media, the method on a second computer comprising:

receiving a group or hash values and a physical location of a data block on a storage media corresponding to each of the hash values, the storage media communicatively coupled to a first computer being reviewed for portions of data;

in response to receiving the group of hash values, for each hash value in the group of hash values performing;

a) looking up a next hash value in a database;

b) in response to the next hash value matching in the database, determining if the hash value that matches is unique to a set of target data file values,

c) in response to the hash value being unique to a set of target data file values, updating metrics for every data file that match the target data file values, otherwise returning to step a;

d) determining if there are more hash values to be examined in the group of hash values

e) in response to more hash values being available, returning to step a, otherwise proceeding to step f;

f) selecting target data files with a highest percentage of hash values that match the target data file values;

g) remove hash values that match across all target data files and determining if any the remaining number of hash values pass a threshold; and

h) in response to the remaining number of target hashes being above a threshold, return to step f, otherwise sending results data to the first computer.

16. A system for identifying portions of data on a storage media, the system comprising:

a computer memory capable of storing machine instructions; and

a hardware processor in communication with the computer memory, the hardware processor configured to access the computer memory, the hardware processor performing

accessing a storage media commutatively coupled to a first computer;

determining if the storage media has a block size with a multiple of 4096 bytes;

in response to the storage media having a block size that is a multiple of 4096 bytes performing;

a) retrieving a next 4096 byte cluster and position;

b) using a first hash function on the 4096 byte cluster to produce a first hash value;

c) applying a first bloom filter to the first hash value;

d) in response to the first bloom filter returning a possibility of the first hash value in a first set of data, performing

e) using a second hash function on the 4096 byte cluster to produce a second hash value;

f) applying a second bloom filter to the second hash value;

g) in response to the second bloom filter returning a possibility of the second hash value in a second set of data, transmitting the second hash value and the position to a second computer;

h) determining whether there are more 4096 byte clusters to be examined; and

.

i) in response to more 4096 byte clusters to be examined, returning to step a.

17. A non-transitory computer program product tangibly embodying computer readable instructions which, when implemented, cause a computer to carry out the steps of a method for identifying portions of data on a storage media, comprising:

accessing a storage media commutatively coupled to a first computer;

determining if the storage media has a block size with a multiple of 4096 bytes;

in response to the storage media having a block size that is a multiple of 4096 bytes performing;

a) retrieving a next 4096 byte cluster and position;

b) using a first hash function on the 4096 byte cluster to produce a first hash value;

c) applying a first bloom filter to the first hash value;

d) in response to the first bloom filter returning a possibility of the first hash value in a first set of data, performing

e) using a second hash function on the 4096 byte cluster to produce a second hash value;

f) applying a second bloom filter to the second hash value;

g) in response to the second bloom filter returning a possibility of the second hash value in a second set of data, transmitting the second hash value and the position to a second computer;

h) determining whether there are more 4096 byte cluster to be examined; and

.

i) in response to more 4096 byte cluster to be examined, returning to step a.

18. The non-transitory computer program product of claim 17, wherein the second bloom filter returning the possibility of the second hash value in the second set of data, comparing the second hash value against a blocked list, and transmitting the second hash value and the position to the second computer in response to the second hash value not in the blocked list.

19. The non-transitory computer program product of claim 18, wherein in response to the second bloom filter returning a possibility of the second hash value in the second set of data, caching the second hash value and the position in a group until one or more of:

all the 4096 byte clusters are examined,

a period of time has expired, or

a threshold of a cache size has been reached,

and then transmitting the group to the second computer.

20. The non-transitory computer program product of claim 19, further comprising:

receiving results data of matching the group to a database of second hash values and physical block identifiers.