Systems and methods for distributive data storage

• US 10,776,014 B2
• Filed: 02/18/2019
• Issued: 09/15/2020
• Est. Priority Date: 01/04/2017
• Status: Active Grant

• Current Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Original Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Inventors: Sehgal, Charandeep, Bhat, Vikas, Krishnan, Ganesh, Kandaswamy, Venkatesh
• Primary Examiner(s): Padmanabhan, Mano
• Assistant Examiner(s): Edouard, Jean C
• Application Number:
  US16/278,327
• Publication Number:
  US 20190179542A1
• Time in Patent Office: 575 Days
• Field of Search: 711162
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Some embodiments can include a system. In some embodiments, a system can comprise one or more processors and one or more non-transitory storage devices storing computing instructions configured to run on the one or more processors and perform acts. In many embodiments, the acts can comprise dividing a computer file into two or more portions; calculating a portion hash value for each portion of the two or more portions, wherein each portion hash value for each portion of the two or more portions can be different than other portion hash values for other portions of the two or more portions; creating two or more computer file portion replicas of each portion of the two or more portions, wherein each of the two or more computer file portion replicas for a corresponding portion of the two or more portions can comprise a same portion hash value as the corresponding portion of the two or more portions; sending the two or more computer file portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes; storing each of the two or more computer file portion replicas in the two or more storage nodes. Other embodiments are disclosed herein.

Claims

1. A system comprising:

one or more processors; and

one or more non-transitory storage devices storing computing instructions configured to run on the one or more processors and perform;

dividing a computer file into two or more portions;

calculating a portion hash value for each respective portion of the two or more portions, wherein each respective portion hash value for each respective portion of the two or more portions is different than other respective portion hash values for other respective portions of the two or more portions;

creating two or more respective computer file portion replicas of each respective portion of the two or more portions, wherein each respective computer file portion replica of the two or more respective computer file portion replicas;

is for a corresponding portion of the two or more portions; and

comprises a same portion hash value as the corresponding portion of the two or more portions;

sending the two or more respective computer file portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes; and

storing each respective computer file portion replica of the two or more respective computer file portion replicas in the two or more storage nodes.

2. The system of claim 1, wherein:

dividing the computer file into the two or more portions comprises determining that a computer file size of the computer file is greater than 2 megabytes.

3. The system of claim 2, wherein:

a respective portion size of each respective portion of the two or more portions is less than 2 megabytes.

4. The system of claim 1, wherein the one or more non-transitory storage devices storing the computing instructions are further configured to run on the one or more processors and perform:

receiving, from an electronic device of a user, a request to utilize the computer file;

sending a request to retrieve the computer file from the distributive data storage cluster;

retrieving the computer file from the distributive data storage cluster by retrieving a set of the two or more respective computer file portion replicas, wherein the set of the two or more respective computer file portion replicas, when combined, comprises the computer file; and

transmitting instructions allowing the electronic device of the user to utilize the computer file.

5. The system of claim 4, wherein the one or more non-transitory storage devices storing the computing instructions are further configured to run on the one or more processors and perform:

calculating a respective retrieved portion hash value for each respective portion of the two or more portions associated with the set of the two or more respective computer file portion replicas, as retrieved; and

determining that the respective retrieved portion hash value for each respective portion of the two or more portions associated with the set of the two or more respective computer file portion replicas, as retrieved, matches the respective portion hash value for each respective portion of the two or more portions, thereby confirming that the two or more respective computer file portion replicas, as retrieved, have not been corrupted.

6. The system of claim 4, wherein the one or more non-transitory storage devices storing the computing instructions are further configured to run on the one or more processors and perform:

merging the set of the two or more respective computer file portion replicas, as retrieved, into the computer file.

7. The system of claim 4, wherein:

at least one respective computer file portion replica of the two or more respective computer file portion replicas, as retrieved, comprise a fastest responding computer file portion replica.

8. The system of claim 1, wherein:

the distributive data storage cluster comprises three storage nodes;

the two or more respective computer file portion replicas comprise three respective computer file portion replicas; and

storing each respective computer file portion replica of the two or more respective computer file portion replicas comprises;

storing one respective computer file portion replica of the three respective computer file portion replicas in each of the three storage nodes.

9. The system of claim 1, wherein:

the computer file comprises an image file.

10. The system of claim 1, wherein creating the two or more respective computer file portion replicas of each respective portion of the two or more portions comprises:

replicating each respective portion of the two or more portions by a respective replication factor based, at least in part, on a number of the two or more storage nodes in the distributive data storage cluster.

11. A method comprising:

dividing a computer file into two or more portions;

calculating a portion hash value for each respective portion of the two or more portions, wherein each respective portion hash value for each respective portion of the two or more portions is different than other respective portion hash values for other respective portions of the two or more portions;

creating two or more respective computer file portion replicas of each respective portion of the two or more portions, wherein each respective computer file portion replica of the two or more respective computer file portion replicas;

is for a corresponding portion of the two or more portions; and

comprises a same portion hash value as the corresponding portion of the two or more portions;

sending the two or more respective computer file portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes; and

storing each respective computer file portion replica of the two or more respective computer file portion replicas in the two or more storage nodes.

12. The method of claim 11, wherein:

dividing the computer file into the two or more portions comprises determining that a computer file size of the computer file is greater than 2 megabytes.

13. The method of claim 12, wherein:

a respective portion size of each respective portion of the two or more portions is less than 2 megabytes.

14. The method of claim 11 further comprising:

receiving, from an electronic device of a user, a request to utilize the computer file;

sending a request to retrieve the computer file from the distributive data storage cluster;

retrieving the computer file from the distributive data storage cluster by retrieving a set of the two or more respective computer file portion replicas, wherein the set of the two or more respective computer file portion replicas, when combined, comprises the computer file; and

transmitting instructions allowing the electronic device of the user to utilize the computer file.

15. The method of claim 14 further comprising:

calculating a respective retrieved portion hash value for each respective portion of the two or more portions associated with the set of the two or more respective computer file portion replicas, as retrieved; and

determining that the respective retrieved portion hash value for each respective portion of the two or more portions associated with the set of the two or more respective computer file portion replicas, as retrieved, matches the respective portion hash value for each respective portion of the two or more portions, thereby confirming that the two or more respective computer file portion replicas, as retrieved, have not been corrupted.

16. The method of claim 14, further comprising:

merging the set of the two or more respective computer file portion replicas, as retrieved, into the computer file.

17. The method of claim 14, wherein:

at least one respective computer file portion replica of the two or more respective computer file portion replicas, as retrieved, comprise a fastest responding computer file portion replica.

18. The method of claim 11, wherein:

the distributive data storage cluster comprises three storage nodes;

the two or more respective computer file portion replicas comprise three respective computer file portion replicas; and

storing each respective computer file portion replica of the two or more respective computer file portion replicas comprises;

storing one respective computer file portion replica of the three respective computer file portion replicas in each of the three storage nodes.

19. The method of claim 11, wherein:

the computer file comprises an image file.

20. The method of claim 11, wherein creating the two or more respective computer file portion replicas of each respective portion of the two or more portions comprises:

replicating each respective portion of the two or more portions by a respective replication factor based, at least in part, on a number of the two or more storage nodes in the distributive data storage cluster.