Systems and methods for distributive data storage

• US 10,209,901 B2
• Filed: 01/04/2017
• Issued: 02/19/2019
• Est. Priority Date: 01/04/2017
• Status: Active Grant

• Current Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Original Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Inventors: Sehgal, Charandeep, Bhat, Vikas, Krishnan, Ganesh, Kandaswamy, Venkatesh
• Primary Examiner(s): Gu, Shawn X
• Assistant Examiner(s): Edouard, Jean C
• Application Number:
  US15/398,034
• Publication Number:
  US 20180188977A1
• Time in Patent Office: 776 Days
• Field of Search: 711162
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

In some embodiments, a method can comprise dividing an image into two or more portions and calculating a portion hash value for each portion of the two or more portions, each portion hash value for each portion of the two or more portions is different than an other portion hash value for other portions of the two or more portions. The method can further comprise creating one or more image portion replicas of each portion of the two or more portions by replicating each portion of the two or more portions by a replication factor, the one or more image portion replicas comprising the same portion hash value as each portion of the two or more portions. The method can further comprise sending the one or more image portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes, and storing each of the one or more image portion replicas in the two or more storage nodes. Other embodiments of related methods and systems are also provided.

Claims

1. A system for more efficient upload, storage, and retrieval of images, the system comprising:

one or more processing modules; and

one or more non-transitory storage modules storing computing instructions configured to run on the one or more processing modules and perform acts of;

dividing an image into two or more portions;

calculating a portion hash value for each portion of the two or more portions, wherein each portion hash value for each portion of the two or more portions is different than other portion hash values for other portions of the two or more portions;

creating one or more image portion replicas of each portion of the two or more portions by replicating each portion of the two or more portions by a replication factor, the one or more image portion replicas comprising a same portion hash value as each corresponding portion of the two or more portions;

sending the one or more image portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes;

storing each of the one or more image portion replicas in the two or more storage nodes, thereby resulting in the more efficient upload, storage, and retrieval of the image;

receiving, from an electronic device of a user, a request to view the image;

sending a request to retrieve the image from the distributive data storage cluster;

retrieving the image from the distributive data storage cluster by retrieving a set of the one or more image portion replicas, wherein the set of the one or more image portion replicas, when combined, comprises the image, thereby resulting in the more efficient retrieval of the image; and

transmitting instructions to display, on the electronic device of the user, the image.

2. The system of claim 1, wherein:

dividing the image into the two or more portions comprises determining that an image size of the image is greater than 2 megabytes.

3. The system of claim 2, wherein:

a portion size of each portion of the two or more portions is less than 2 megabytes.

4. The system of claim 1, wherein the one or more non-transitory storage modules storing the computing instructions are configured to run on the one or more processing modules and further perform acts of:

calculating a retrieved portion hash value for each portion of the two or more portions associated with the set of the one or more image portion replicas retrieved; and

determining that the retrieved portion hash value for each portion of the two or more portions associated with the set of the one or more image portion replicas retrieved matches the portion hash value for each portion of the two or more portions, thereby confirming that the one or more image portion replicas retrieved have not been corrupted.

5. The system of claim 1, wherein the one or more non-transitory storage modules storing the computing instructions are configured to run on the one or more processing modules and further perform an act of:

merging the set of the one or more image portion replicas into the image.

6. The system of claim 1, wherein:

each of the one or more image portion replicas of the set of the one or more image portion replicas comprise a fastest responding image portion replica.

7. The system of claim 1, wherein:

the distributive data storage cluster comprises three storage nodes.

8. The system of claim 1, wherein:

the replication factor is based at least in part on a number of the two or more storage nodes in the distributive data storage cluster.

9. A method for more efficient upload, storage, and retrieval of images, the method comprising:

dividing an image into two or more portions;

calculating a portion hash value for each portion of the two or more portions, wherein each portion hash value for each portion of the two or more portions is different than other portion hash values for other portions of the two or more portions;

creating one or more image portion replicas of each portion of the two or more portions by replicating each portion of the two or more portions by a replication factor, the one or more image portion replicas comprising a same portion hash value as each corresponding portion of the two or more portions;

sending the one or more image portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes;

storing each of the one or more image portion replicas in the two or more storage nodes, thereby resulting in the more efficient upload, storage, and retrieval of the image;

receiving, from an electronic device of a user, a request to view the image;

sending a request to retrieve the image from the distributive data storage cluster;

retrieving the image from the distributive data storage cluster by retrieving a set of the one or more image portion replicas, wherein the set of the one or more image portion replicas, when combined, comprises the image, thereby resulting in the more efficient retrieval of the image; and

transmitting instructions to display, on the electronic device of the user, the image.

10. The method of claim 9, wherein:

dividing the image into the two or more portions comprises determining that an image size of the image is greater than 2 megabytes.

11. The method of claim 9, further comprising:

calculating a retrieved portion hash value for each portion of the two or more portions associated with the set of the one or more image portion replicas retrieved; and

determining that the retrieved portion hash value for each portion of the two or more portions associated with the set of the one or more image portion replicas retrieved matches the portion hash value for each portion of the two or more portions, thereby confirming that the one or more image portion replicas retrieved have not been corrupted.

12. The method of claim 9, further comprising:

merging the set of the one or more image portion replicas into the image.

13. The method of claim 9, wherein:

each of the one or more image portion replicas of the set of the one or more image portion replicas comprise a fastest responding image portion replica.

14. The method of claim 9, wherein:

the distributive data storage cluster comprises three storage nodes.

15. The method of claim 9, wherein:

the replication factor is based at least in part on a number of the two or more storage nodes in the distributive data storage cluster.

16. A method for more efficient upload, storage, and retrieval of images, the method comprising:

dividing an image into at least a first portion and a second portion;

calculating a first portion hash value for the first portion and a second portion hash value for the second portion, wherein the first portion hash value and the second portion hash value are different from each other;

creating one or more first image portion replicas of the first portion and one or more second image portion replicas of the second portion, wherein;

a number of the one or more first image portion replicas and the one or more second image portion replicas are based at least in part on a replication factor;

the one or more first image portion replicas comprise the first portion hash value; and

the one or more second image portion replicas comprise the second portion hash value;

sending the one or more first image portion replicas and the one or more second image portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes;

storing each of the one or more first image portion replicas and the one or more second image portion replicas in the two or more storage nodes, thereby resulting in the more efficient retrieval of the image;

receiving, from an electronic device of a user, a request to view the image;

sending a request to retrieve the image from the distributive data storage cluster;

retrieving the image from the distributive data storage cluster by retrieving one of the one or more first image portion replicas and one of the one or more second image portion replicas, thereby resulting in the more efficient retrieval of the image; and

transmitting instructions to display, on the electronic device of the user, the image.

17. The method of claim 16, wherein:

dividing the image into at least the first portion and the second portion comprises determining that an image size of the image is greater than 2 megabytes.

18. The method of claim 16, further comprising:

calculating a retrieved first portion hash value for the one of the one or more first image portion replicas retrieved, and a retrieved second portion hash value for the one of the one or more second image portion replicas retrieved; and

determining that the retrieved first portion hash value matches the first portion hash value and that the retrieved second portion hash value matches the second portion hash value, thereby confirming that one or more image portion replicas retrieved have not been corrupted.

19. The method of claim 16, further comprising:

merging the one of the one or more first image portion replicas and the one of the one or more second image portion replicas into the image.

20. The method of claim 16, wherein:

the one of the one or more first image portion replicas comprises a fastest responding first image portion replica of the one or more first image portion replicas and the one of the one or more second image portion replicas comprises a fastest responding second image portion replica of the one or more second image portion replicas.