Developing an accurate dispersed storage network memory performance model through training

• US 10,089,178 B2
• Filed: 01/04/2017
• Issued: 10/02/2018
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Iljazi, Ilir
• Primary Examiner(s): Chaudry, Mujtaba M
• Application Number:
  US15/398,540
• Publication Number:
  US 20170249551A1
• Time in Patent Office: 636 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/327 Alarm or error message display
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0629 : Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/064 : Management of blocks
• CPC Class Codes: G06F 3/0644 : Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0653 : Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 : Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 30/20 : Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06N 3/04 : Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/10 : Interfaces, programming languages or software development kits, e.g. for simulating neural networks
• CPC Class Codes: G06Q 10/063116 : Schedule adjustment for a person or group
• CPC Class Codes: G06Q 10/06316 : Sequencing of tasks or work
• CPC Class Codes: G06Q 10/20 : Administration of product repair or maintenance
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H03M 13/2909 : Product codes
• CPC Class Codes: H03M 13/3761 : using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/616 : Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 63/0428 : wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/061 : for key exchange, e.g. in peer-to-peer networks cryptographic mechanisms or cryptographic arrangements for key agreement H04L9/0838
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 9/0869 : involving random numbers or seeds
• CPC Class Codes: H04L 9/0894 : Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage
• CPC Class Codes: H04L 9/14 : using a plurality of keys or algorithms
• CPC Class Codes: H04L 9/3242 : involving keyed hash functions, e.g. message authentication codes [MACs], CBC-MAC or HMAC

Abstract

A computing device includes an interface configured to interface and communicate with a dispersed or distributed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. The computing device receives first samples corresponding to inputs that characterize configuration of the DSN and receives second samples corresponding to outputs that characterize system behavior of the DSN. The computing device then processes the first and samples to generate a DSN model to generate predictive performance of the outputs based on various values of the inputs. In some instances, the DSN model is based on a neural network model that employs the inputs that characterize the configuration of the DSN and generates the outputs that characterize system behavior of the DSN.

Claims

1. A computing device comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

receive a first plurality of samples corresponding to a plurality of inputs that characterize configuration of the DSN;

receive a second plurality of samples corresponding to a plurality of outputs that characterize system behavior of the DSN; and

process the first plurality of samples and the second plurality of samples to generate a DSN model to generate predictive performance of the plurality of outputs based on various values of the plurality of inputs that characterize the configuration of the DSN.

2. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

generate the DSN model based on a neural network model that includes the plurality of inputs that characterize the configuration of the DSN as inputs of the neural network model and the plurality of outputs that characterize system behavior of the DSN as outputs of the neural network model, and a plurality of hidden neurons of at least one layer that receive the inputs of the neural network model and generate the outputs of the neural network model, wherein connections from the plurality of hidden neurons to other hidden neurons or the outputs of the neural network model are weighted based on a plurality of weights;

initialize the plurality of weights to a predetermined plurality of values; and

adjust the plurality of weights from the predetermined plurality of values based on backpropagation of error values of the neural network model backwards through the neural network model starting at the outputs of the neural network model so that each hidden neuron of the plurality of hidden neurons includes a corresponding error value that substantially represents its respective contribution to outputs of the neural network model, wherein the error values of the neural network model correspond to differences between the outputs of the neural network model and expected outputs of the neural network model.

3. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

receive a third plurality of samples corresponding to the plurality of inputs that characterize the configuration of the DSN after receiving the first plurality of samples corresponding to the plurality of inputs that characterize configuration of the DSN;

receive a fourth plurality of samples corresponding to the plurality of outputs that characterize system behavior of the DSN after receiving the second plurality of samples corresponding to the plurality of outputs that characterize system behavior of the DSN; and

process the third plurality of samples and the fourth plurality of samples to update the DSN model to update the predictive performance of the plurality of outputs based on various values of the plurality of inputs that characterize the configuration of the DSN.

4. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, further comprises:

a modeling module configured to generate DSN behavioral model information based on the plurality of inputs that characterize configuration of the DSN; and

a simulation module configured to generate the predictive performance of the plurality of outputs, wherein the simulation module is configured to feedback the predictive performance of the plurality of outputs to the modeling module for use by the modeling module to generate subsequent DSN behavioral model information based on the plurality of inputs that characterize configuration of the DSN and the predictive performance of the plurality of outputs.

5. The computing device of claim 1, wherein the plurality of inputs that characterize the configuration of the DSN include at least one of:

a distribution of a data object within the DSN, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of encoded data slices (EDSs) that is of pillar width, wherein the set of EDSs are distributedly stored among a plurality of storage units (SUs) of the DSN, wherein a decode threshold number of EDSs are needed to recover the data segment, wherein a read threshold number of EDSs provides for reconstruction of the data segment, and wherein a write threshold number of EDSs provides for a successful transfer of the set of EDSs from a first at least one location in the DSN to a second at least one location in the DSN;

a concurrency level of storage of the data object within the DSN including storage of copies of at least the read threshold number of EDSs corresponding to the data object within the DSN;

a number of dispersed storage (DS) units and types of DS units within the DSN;

a number of network communication links and latencies of network communication links within the DSN;

a configuration of an information dispersal algorithm (IDA) including the dispersed error encoding parameters employed within the DSN;

one or more codecs employed within the DSN;

or

one or more network security protocols employed within the DSN.

6. The computing device of claim 1, wherein the plurality of outputs that characterize system behavior of the DSN include at least one of:

a first number of sustained operations including data access requests per second per dispersed storage (DS) unit within the DSN;

a second number of sustained operations including data access requests per second per storage unit (SU) within the DSN;

a first throughput level per DS unit within the DSN;

a second throughput level per SU within the DSN;

a first latency from at least one SU to at least one DS processing unit within the DSN;

or

a second latency from at least one computing device providing at least one data access request to the at least one DS processing unit within the DSN.

7. The computing device of claim 1 further comprising:

a storage unit (SU) of a plurality of storage units (SUs) within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

8. The computing device of claim 1, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

9. A method for execution by a computing device, the method comprising:

receiving, via an interface of the computing device that is configured to interface and communicate with a dispersed or distributed storage network (DSN), a first plurality of samples corresponding to a plurality of inputs that characterize configuration of the DSN;

receiving, via the interface of the computing device, a second plurality of samples corresponding to a plurality of outputs that characterize system behavior of the DSN; and

processing the first plurality of samples and the second plurality of samples to generate a DSN model to generate predictive performance of the plurality of outputs based on various values of the plurality of inputs that characterize the configuration of the DSN.

10. The method of claim 9 further comprising:

generating the DSN model based on a neural network model that includes the plurality of inputs that characterize configuration of the DSN as inputs of the neural network model and the plurality of outputs that characterize system behavior of the DSN as outputs of the neural network model, and a plurality of hidden neurons of at least one layer that receive the inputs of the neural network model and generate the outputs of the neural network model, wherein connections from the plurality of hidden neurons to other hidden neurons or the outputs of the neural network model are weighted based on a plurality of weights;

initializing the plurality of weights to a predetermined plurality of values; and

adjusting the plurality of weights from the predetermined plurality of values based on backpropagation of error values of the neural network model backwards through the neural network model starting at the outputs of the neural network model so that each hidden neuron of the plurality of hidden neurons includes a corresponding error value that substantially represents its respective contribution to outputs of the neural network model, wherein the error values of the neural network model correspond to differences between the outputs of the neural network model and expected outputs of the neural network model.

11. The method of claim 9 further comprising:

receiving, via the interface of the computing device, a third plurality of samples corresponding to the plurality of inputs that characterize configuration of the DSN after receiving the first plurality of samples corresponding to the plurality of inputs that characterize the configuration of the DSN;

receiving, via the interface of the computing device, a fourth plurality of samples corresponding to the plurality of outputs that characterize system behavior of the DSN after receiving the second plurality of samples corresponding to the plurality of outputs that characterize system behavior of the DSN; and

processing the third plurality of samples and the fourth plurality of samples to update the DSN model to update the predictive performance of the plurality of outputs based on various values of the plurality of inputs that characterize the configuration of the DSN.

12. The method of claim 9 further comprising:

operating a modeling module of the computing device to generate DSN behavioral model information based on the plurality of inputs that characterize the configuration of the DSN; and

operating a simulation module of the computing device to generate the predictive performance of the plurality of outputs, wherein the simulation module is configured to feedback the predictive performance of the plurality of outputs to the modeling module for use by the modeling module to generate subsequent DSN behavioral model information based on the plurality of inputs that characterize the configuration of the DSN and the predictive performance of the plurality of outputs.

13. The method of claim 9, wherein the plurality of inputs that characterize the configuration of the DSN include at least one of:

a distribution of a data object within the DSN, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of encoded data slices (EDSs) that is of pillar width, wherein the set of EDSs are distributedly stored among a plurality of storage units (SUs) of the DSN, wherein a decode threshold number of EDSs are needed to recover the data segment, wherein a read threshold number of EDSs provides for reconstruction of the data segment, and wherein a write threshold number of EDSs provides for a successful transfer of the set of EDSs from a first at least one location in the DSN to a second at least one location in the DSN;

a concurrency level of storage of the data object within the DSN including storage of copies of at least the read threshold number of EDSs corresponding to the data object within the DSN;

a number of dispersed storage (DS) units and types of DS units within the DSN;

a number of network communication links and latencies of network communication links within the DSN;

a configuration of an information dispersal algorithm (IDA) including the dispersed error encoding parameters employed within the DSN;

one or more codecs employed within the DSN;

or

one or more network security protocols employed within the DSN.

14. The method of claim 9, wherein the plurality of outputs that characterize system behavior of the DSN include at least one of:

a first number of sustained operations including data access requests per second per dispersed storage (DS) unit within the DSN;

a second number of sustained operations including data access requests per second per storage unit (SU) within the DSN;

a first throughput level per DS unit within the DSN;

a second throughput level per SU within the DSN;

a first latency from at least one SU to at least one DS processing unit within the DSN;

or

a second latency from at least one computing device providing at least one data access request to the at least one DS processing unit within the DSN.

15. The method of claim 9, wherein:

the computing device is a SU of the plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device; and

the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).