Data transfer reduction in scale out architectures
First Claim
1. A method, comprising:
- receiving a data stream at a compute node, the compute node having compute node local storage resources;
separating the data stream into a plurality of chunks, including a first chunk and a second chunk;
generating a plurality of fingerprints for the plurality of chunks, including the first chunk and the second chunk;
transmitting the plurality of fingerprints to a scale out node, the scale out node having scale out node local storage resources, wherein the scale out node compares the plurality of fingerprints with fingerprints corresponding to chunks maintained in the scale out node local storage resources;
creating an object map for the data stream based on a determination of whether fingerprints in the plurality of fingerprints correspond to fingerprints already stored in the scale out node; and
creating a datastore suitcase corresponding to the object map including an index portion and a data portion, the data portion holding a plurality of datastore indices corresponding to the plurality of chunks and a last file entry for each of the plurality of chunks, each last file entry storing an identifier of a file which last placed a reference to the corresponding chunk.
16 Assignments
0 Petitions
Accused Products
Abstract
Mechanisms are provided for data transfer reduction in scale out architectures. When a compute node receives a write input/output (I/O) request for a data stream, the compute node separates the data stream into chunks and generates fingerprints for the individual chunks. Fingerprints are then sent to a scale out node and compared to fingerprints of chunks already maintained at the scale out node. Write data transfers are only made for chunks not already maintained at the scale out node. For a read I/O request for a data stream, fingerprints for chunks of the data stream are requested by the compute node from a scale out node. Fingerprints received are compared to fingerprints of chunks already maintained at the compute node and read data transfers are only made for chunks not already maintained at the compute node.
14 Citations
22 Claims
-
1. A method, comprising:
-
receiving a data stream at a compute node, the compute node having compute node local storage resources; separating the data stream into a plurality of chunks, including a first chunk and a second chunk; generating a plurality of fingerprints for the plurality of chunks, including the first chunk and the second chunk; transmitting the plurality of fingerprints to a scale out node, the scale out node having scale out node local storage resources, wherein the scale out node compares the plurality of fingerprints with fingerprints corresponding to chunks maintained in the scale out node local storage resources; creating an object map for the data stream based on a determination of whether fingerprints in the plurality of fingerprints correspond to fingerprints already stored in the scale out node; and creating a datastore suitcase corresponding to the object map including an index portion and a data portion, the data portion holding a plurality of datastore indices corresponding to the plurality of chunks and a last file entry for each of the plurality of chunks, each last file entry storing an identifier of a file which last placed a reference to the corresponding chunk. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A system, comprising:
-
an interface configured to receive a data stream at a compute node, the compute node having compute node local storage resources; a processor configured to separate the data stream into a plurality of chunks and generate a plurality of fingerprints for the plurality of chunks, the plurality of chunks including a first chunk and a second chunk; wherein the plurality of fingerprints are transmitted to a scale out node, the scale out node having scale out node local storage resources, wherein the scale out node compares the plurality of fingerprints with fingerprints corresponding to chunks maintained in the scale out node local storage resources; wherein the processor is further configured to create an object map for the data stream based on a determination of whether fingerprints in the plurality of fingerprints correspond to fingerprints already stored in the scale out node; and wherein the processor is further configured to create a datastore suitcase corresponding to the object map including an index portion and a data portion, the data portion holding a plurality of datastore indices corresponding to the plurality of chunks and a last file entry for each of the plurality of chunks, each last file entry storing an identifier of a file which last placed a reference to the corresponding chunk. - View Dependent Claims (10, 11, 12, 13, 14, 15, 16)
-
-
17. A method, comprising:
-
receiving a read input/output (I/O) request for a data stream at a compute node, the compute node having compute node local storage resources; identifying a scale out node having an object map corresponding to the data stream, the object map further corresponding to a datastore suitcase including an index portion and a data portion, the data portion holding a plurality of datastore indices corresponding to a plurality of chunks, the chunks corresponding to the data stream, and a last file entry for each of the plurality of chunks, each last file entry storing an identifier of a file which last placed a reference to the corresponding chunk; requesting a plurality of fingerprints for the data stream from the scale out node; comparing the plurality of fingerprints with fingerprints corresponding to chunks maintained in compute node local storage resources; requesting from the scale out node only the chunks not maintained in the compute node local storage resources. - View Dependent Claims (18, 19, 20, 21, 22)
-
Specification