Ensuring consistency between content and metadata with intents

• US 10,241,677 B2
• Filed: 02/24/2017
• Issued: 03/26/2019
• Est. Priority Date: 02/24/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dubucq, Thomas, Jarczyk, Jeremy S., Khadiwala, Ravi V., Motwani, Manish, Narubin, Renars W., Resch, Jason K., Scholl, Daniel J., Vedpathak, Yogesh R.
• Primary Examiner(s): Chase, Shelly A
• Application Number:
  US15/442,240
• Publication Number:
  US 20180246644A1
• Time in Patent Office: 760 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/616 Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations

Abstract

A storage unit (SU) includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the SU based on the operational instructions, is configured to perform various operations. The SU stores at least one encoded data slice (EDS) of first EDSs corresponding to a data object that are distributedly stored in first SUs and also an intent message that includes specifications for consistency between the data object and metadata of the data object. A second set of EDSs corresponding to the metadata are distributedly stored in second SUs. The SU services the intent message to determine consistency of the data object and the metadata based on the specifications and deletes the intent message when they are consistent.

Claims

1. A storage unit (SU) comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the SU based on the operational instructions, is configured to;

store, based on a first write command that is received via the DSN and via the interface from a computing device, at least one encoded data slice (EDS) of a first set of encoded data slices (EDSs) corresponding to a data object that are distributedly stored in a first plurality of SUs that includes the SU within the DSN, wherein the data object is segmented into a first plurality of data segments, wherein a data segment of the first plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the first set of EDSs;

store, based on a second write command that is received via the DSN and via the interface from the computing device or another computing device, an intent message that includes specifications for consistency between the data object and metadata of the data object, wherein a second set of EDSs corresponding to the metadata of the data object are distributedly stored in a second plurality of SUs within the DSN, wherein the metadata is segmented into a second plurality of data segments, wherein a data segment of the second plurality of data segments is dispersed error encoded in accordance with the dispersed error encoding parameters or other dispersed error encoding parameters to produce the second set of EDSs;

service the intent message to determine whether the data object and the metadata of the data object are consistent based on the specifications;

when the data object and the metadata of the data object are determined to be consistent based on the service the intent message, delete the intent message that is stored in the SU; and

when the data object and the metadata of the data object are determined to be inconsistent based on the service the intent message, eliminate inconsistency between the data object and the metadata of the data object within the DSN.

2. The SU of claim 1, wherein the processing module, when operable within the SU based on the operational instructions, is further configured to:

determine a time at or during which to service the intent message; and

eliminate the inconsistency between the data object and the metadata of the data object by;

transmitting a delete command via the DSN for at least one of the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs or the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs;

transmitting a first repair command via the DSN to repair the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs to be consistent with the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs;

or

transmitting a second repair command via the DSN to repair the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs to be consistent with the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs.

3. The SU of claim 1, wherein the processing module, when operable within the SU based on the operational instructions and when the data object and the metadata of the data object are determined to be inconsistent based on the service the intent message, is further configured to:

transmit a notification of inconsistency between the data object and the metadata of the data object to the computing device, wherein the computing device is associated with a user;

receive an instruction message from the computing device associated with the user that specifies an operation to be performed by the SU to eliminate the inconsistency between the data object and the metadata of the data object; and

perform the operation specified in the instruction message to eliminate the inconsistency between the data object and the metadata of the data object.

4. The SU of claim 1, wherein the processing module, when operable within the SU based on the operational instructions, is further configured to:

determine the data object and the metadata of the data object are inconsistent based on the specifications when a commit phase or a finalize phase of a 3 phase write process that includes a write phase, the commit phase, and the finalize phase for storage of the first set of EDSs corresponding to the data object within the first plurality of SUs that includes the SU within the DSN or the second set of EDSs corresponding to the metadata of the data object within the second plurality of SUs within the DSN is determined to have failed.

5. The SU of claim 1, wherein the processing module, when operable within the SU based on the operational instructions, is further configured to:

determine, when servicing the intent message to determine whether the data object and the metadata of the data object are consistent based on the specifications, that the data object and the metadata of the data object are consistent when a first version of the at least one EDS of the first set of EDSs corresponding to the data object compares favorably with a second version of the second set of EDSs corresponding to the metadata of the data object;

or

determine, when servicing the intent message to determine whether the data object and the metadata of the data object are consistent based on the specifications, that the data object and the metadata of the data object are consistent when a first system configuration of a Decentralized, or Distributed, Agreement Protocol (DAP) of the DSN that corresponds to storage of the at least one EDS of the first set of EDSs corresponding to the data object compares favorably with a second system configuration of the DAP of the DSN that corresponds to storage of the second set of EDSs corresponding to the metadata of the data object.

6. The SU of claim 1, wherein the SU is located at a first premises that is remotely located from at least one SU of at least one SU of the first plurality of SUs or the second plurality of SUs within the DSN.

7. The SU of claim 1, wherein the computing device includes another SU of the first plurality of SUs, an SU of the second plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

8. The SU of claim 1, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

9. A storage unit (SU) comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the SU based on the operational instructions, is configured to;

store, based on a first write command that is received via the DSN and via the interface from a computing device, at least one encoded data slice (EDS) of a first set of encoded data slices (EDSs) corresponding to a data object that are distributedly stored in a first plurality of SUs that includes the SU within the DSN, wherein the data object is segmented into a first plurality of data segments, wherein a data segment of the first plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the first set of EDSs;

store, based on a second write command that is received via the DSN and via the interface from the computing device or another computing device, an intent message that includes specifications for consistency between the data object and metadata of the data object, wherein a second set of EDSs corresponding to the metadata of the data object are distributedly stored in a second plurality of SUs within the DSN, wherein the metadata is segmented into a second plurality of data segments, wherein a data segment of the second plurality of data segments is dispersed error encoded in accordance with the dispersed error encoding parameters or other dispersed error encoding parameters to produce the second set of EDSs;

determine a time at or during which to service the intent message;

service the intent message at or during the time to determine whether the data object and the metadata of the data object are consistent based on the specifications including to determine that the data object and the metadata of the data object are inconsistent based on the specifications when a commit phase or a finalize phase of a 3 phase write process that includes a write phase, the commit phase, and the finalize phase for storage of the first set of EDSs corresponding to the data object within the first plurality of SUs that includes the SU within the DSN or the second set of EDSs corresponding to the metadata of the data object within the second plurality of SUs within the DSN is determined to have failed;

when the data object and the metadata of the data object are determined to be consistent based on the service the intent message, delete the intent message that is stored in the SU; and

when the data object and the metadata of the data object are determined to be inconsistent based on the service the intent message, eliminate inconsistency between the data object and the metadata of the data object within the DSN.

10. The SU of claim 9, wherein the processing module, when operable within the SU based on the operational instructions, is further configured to:

determine a time at or during which to service the intent message; and

eliminate the inconsistency between the data object and the metadata of the data object by;

transmitting a delete command via the DSN for at least one of the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs or the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs;

transmitting a first repair command via the DSN to repair the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs to be consistent with the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs;

or

transmitting a second repair command via the DSN to repair the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs to be consistent with the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs.

11. The SU of claim 9, wherein the processing module, when operable within the SU based on the operational instructions, is further configured to:

determine, when servicing the intent message to determine whether the data object and the metadata of the data object are consistent based on the specifications, that the data object and the metadata of the data object are consistent when a first version of the at least one EDS of the first set of EDSs corresponding to the data object compares favorably with a second version of the second set of EDSs corresponding to the metadata of the data object;

or

determine, when servicing the intent message to determine whether the data object and the metadata of the data object are consistent based on the specifications, that the data object and the metadata of the data object are consistent when a first system configuration of a Decentralized, or Distributed, Agreement Protocol (DAP) of the DSN that corresponds to storage of the at least one EDS of the first set of EDSs corresponding to the data object compares favorably with a second system configuration of the DAP of the DSN that corresponds to storage of the second set of EDSs corresponding to the metadata of the data object.

12. The SU of claim 9, wherein the computing device includes another SU of the first plurality of SUs, an SU of the second plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

13. The SU of claim 9, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

14. A method for execution by a storage unit (SU), the method comprising:

storing, based on a first write command from a computing device that is received via an interface of the SU that is configured to interface and communicate with a dispersed or distributed storage network (DSN), at least one encoded data slice (EDS) of a first set of encoded data slices (EDSs) corresponding to a data object that are distributedly stored in a first plurality of SUs that includes the SU within the DSN, wherein the data object is segmented into a first plurality of data segments, wherein a data segment of the first plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce the first set of EDSs;

storing, based on a second write command from the computing device or another computing device that is received via the interface of the SU, an intent message that includes specifications for consistency between the data object and metadata of the data object, wherein a second set of EDSs corresponding to the metadata of the data object are distributedly stored in a second plurality of SUs within the DSN, wherein the metadata is segmented into a second plurality of data segments, wherein a data segment of the second plurality of data segments is dispersed error encoded in accordance with the dispersed error encoding parameters or other dispersed error encoding parameters to produce the second set of EDSs;

servicing the intent message to determine whether the data object and the metadata of the data object are consistent based on the specifications;

when the data object and the metadata of the data object are determined to be consistent based on the service the intent message, deleting the intent message that is stored in the SU; and

when the data object and the metadata of the data object are determined to be inconsistent based on the service the intent message, eliminating inconsistency between the data object and the metadata of the data object within the DSN.

15. The method of claim 14 further comprising:

determining a time at or during which to service the intent message; and

eliminating the inconsistency between the data object and the metadata of the data object by;

transmitting, via the interface of the SU and via the DSN, a delete command for at least one of the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs or the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs;

transmitting, via the interface of the SU and via the DSN, a first repair command to repair the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs to be consistent with the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs;

or

transmitting, via the interface of the SU and via the DSN, a second repair command to repair the second set of EDSs corresponding to the metadata of the data object that are distributedly stored in the second plurality of SUs to be consistent with the first set of EDSs corresponding to the data object that are distributedly stored in the first plurality of SUs.

16. The method of claim 14, when the data object and the metadata of the data object are determined to be inconsistent based on the service the intent message, further comprising:

transmitting, via the interface of the SU and via the DSN, a notification of inconsistency between the data object and the metadata of the data object to the computing device, wherein the computing device is associated with a user;

receiving, via the interface of the SU and via the DSN, an instruction message from the computing device associated with the user that specifies an operation to be performed by the SU to eliminate the inconsistency between the data object and the metadata of the data object; and

performing the operation specified in the instruction message to eliminate the inconsistency between the data object and the metadata of the data object.

17. The method of claim 14 further comprising:

determining the data object and the metadata of the data object are inconsistent based on the specifications when a commit phase or a finalize phase of a 3 phase write process that includes a write phase, the commit phase, and the finalize phase for storage of the first set of EDSs corresponding to the data object within the first plurality of SUs that includes the SU within the DSN or the second set of EDSs corresponding to the metadata of the data object within the second plurality of SUs within the DSN is determined to have failed.

18. The method of claim 14 further comprising:

determining, when servicing the intent message to determine whether the data object and the metadata of the data object are consistent based on the specifications, that the data object and the metadata of the data object are consistent when a first version of the at least one EDS of the first set of EDSs corresponding to the data object compares favorably with a second version of the second set of EDSs corresponding to the metadata of the data object;

or

determining, when servicing the intent message to determine whether the data object and the metadata of the data object are consistent based on the specifications, that the data object and the metadata of the data object are consistent when a first system configuration of a Decentralized, or Distributed, Agreement Protocol (DAP) of the DSN that corresponds to storage of the at least one EDS of the first set of EDSs corresponding to the data object compares favorably with a second system configuration of the DAP of the DSN that corresponds to storage of the second set of EDSs corresponding to the metadata of the data object.

19. The method of claim 14, wherein:

the SU is located at a first premises that is remotely located from at least one SU of at least one SU of the first plurality of SUs or the second plurality of SUs within the DSN; and

the computing device includes another SU of the first plurality of SUs, an SU of the second plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

20. The method of claim 14, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).