Isolating the introduction of software defects in a dispersed storage network

• US 10,255,133 B2
• Filed: 08/26/2016
• Issued: 04/09/2019
• Est. Priority Date: 03/29/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Algie, Teague S., Peake, Andrew G., Seaborn, Mark D., Volvovski, Ilya
• Primary Examiner(s): Bonzo, Bryce P
• Assistant Examiner(s): Lin, Katherine
• Application Number:
  US15/248,772
• Publication Number:
  US 20170286280A1
• Time in Patent Office: 956 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H03M 13/373 : with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes
• CPC Class Codes: H03M 13/616 : Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A dispersed storage network facilitates isolating the introduction of software defects in dispersed storage units. A search strategy is employed whereby after identifying a test failure in a current version of the memory software code, a code version since a previous successfully tested version is identified. An interim version that represents the point at which approximately one half the changes were introduced is then tested. When there is a test failure, the next interim version selected for testing represents the point at which approximately one half the changes were introduced between the first interim version tested and the current version. If no failure, a next interim version is tested that represents the point at which approximately one half the changes were introduced between the previous successfully tested version and the first interim version tested.

Claims

1. A method of identifying changes in memory software code of a first storage unit of a dispersed storage network (DSN) that includes a plurality of storage units, the method performed by a processing module of the dispersed storage network, comprising:

receiving, from a device of the DSN, an indication of a software operational error resulting from a first test performed on a current version of the memory software code;

in response to receiving the indication of a software operational error, identifying a sequence of versions of the memory software code including a sequence of interim versions of the memory software code between the current version of the memory software code and a previous version of the memory software code associated with favorable software operation;

identifying a first version of the memory software code from the sequence of interim versions of the memory software code, where the first version of the memory software code is in the middle of or approximately in the middle of the sequence of interim versions of the memory software code;

facilitating a second test of the first version of the memory software code by at least one device of the DSN;

determining if the second test results in a test failure;

in response to determining that the second test results in a test failure;

identifying a second version of the memory software code from the sequence of interim versions of the memory software code, where the second version of the memory software code is in the middle of or approximately in the middle of a sub-sequence of the sequence of interim versions of the memory software code between the first version of the memory software code and the previous version of the memory software code associated with favorable software operation; and

facilitating a third test of the second version of the memory software code; and

in response to determining that the second test does not result in a test failure;

identifying a third version of the memory software code from the sequence of interim versions of the memory software code, where the third version of the memory software code is in the middle of or approximately in the middle of a sub-sequence of the sequence of interim versions of the memory software code between the first version of the memory software code and the current version of the memory software code; and

facilitating a fourth test of the third version of the memory software code.

2. The method of claim 1 wherein receiving an indication of a software operational error resulting from the first test performed on the current version of the memory software code includes one or more of interpreting an error message, selecting the first test from a plurality of software tests, facilitating exercise of the memory software code in accordance with the first test, or indicating a software error when a software test resultant compares unfavorably to an expected software test resultant.

3. The method of claim 1 wherein receiving an indication of a software operational error resulting from the first test performed on the current version of the memory software code includes one or more of issuing a test request to the first storage unit or receiving a test response from the first storage unit.

4. The method of claim 3 wherein receiving an indication of a software operational error resulting from the first test performed on the current version of the memory software code includes issuing a test request to a DSN computing device that is configured to subsequently issue first test slices to the first storage unit and interpret second test slices received from the first storage unit.

5. The method of claim 1 wherein identifying the sequence of versions of the memory software code includes interpreting a software test record using an identifier of a software test performed on the previous version of the memory software code associated with favorable software operation.

6. The method of claim 1 wherein determining if the second test results in a test failure includes loading a second storage unit of the DSN with the first version of the memory software code and conducting at least a portion of the second test on the second storage unit.

7. The method of claim 1 wherein the second test is selected from a set of tests that includes the first test.

8. The method of claim 7 wherein the second test includes at least a portion of the first test.

9. The method of claim 7 wherein the second test is selected based, at least in part, on those areas of the memory software code that were changed since the previous version of the memory software code associated with favorable software operation.

10. A dispersed storage network managing unit for use in a dispersed storage network (DSN), the DSN including a plurality of storage units, the dispersed storage network managing unit comprising:

a communications interface;

a memory;

a computer processor;

where the memory includes instructions for causing the computer processor to;

receive, from a device of the DSN via the communications interface, an indication of a software operational error resulting from a first test performed on a current version of a memory software code;

in response to receiving the indication of a software operational error, identify a sequence of versions of the memory software code including a sequence of interim versions of the memory software code between the current version of the memory software code and a previous version of the memory software code associated with favorable software operation;

identify a first version of the memory software code from the sequence of interim versions of the memory software code, where the first version of the memory software code is in the middle of or approximately in the middle of the sequence of interim versions of the memory software code;

facilitate a second test of the first version of the memory software code by at least one device of the DSN;

determine if the second test results in a test failure;

in response to determining that the second test results in a test failure;

identify a second version of the memory software code from the sequence of interim versions of the memory software code, where the second version of the memory software code is in the middle of or approximately in the middle of a sub-sequence of the sequence of interim versions of the memory software code between the first version of the memory software code and the previous version of the memory software code associated with favorable software operation; and

facilitate a third test of the second version of the memory software code; and

in response to determining that the second test does not result in a test failure;

identify a third version of the memory software code from the sequence of interim versions of the memory software code, where the third version of the memory software code is in the middle of or approximately in the middle of a sub-sequence of the sequence of interim versions of the memory software code between the first version of the memory software code and the current version of the memory software code; and

facilitate a fourth test of the third version of the memory software code.

11. The dispersed storage network managing unit of claim 10 wherein the memory further includes instructions for causing the computer processor to receive an indication of a software operational error resulting from the first test performed on the current version of the memory software code by performing one or more of interpreting an error message, selecting the first test from a plurality of software tests, facilitating exercise of the memory software code in accordance with the first test, or indicating a software error when a software test resultant compares unfavorably to an expected software test resultant.

12. The dispersed storage network managing unit of claim 10 wherein the memory further includes instructions for causing the computer processor to receive an indication of a software operational error resulting from the first test performed on the current version of the memory software code by issuing a test request to a first storage unit of the plurality of storage units and receiving a test response from the first storage unit.

13. The dispersed storage network managing unit of claim 10 communicatively coupled to a dispersed storage network computing device comprising:

a second communications interface;

a second memory; and

a second computer processor;

where the second memory includes instructions for causing the second computer processor to;

receive a test request from the dispersed storage network managing unit via the second communications interface; and

in response to receipt of the test request, issue first test slices to a first storage unit of the plurality of storage units and interpret second test slices received from the first storage unit.

14. The dispersed storage network managing unit of claim 10 wherein the memory further includes instructions for causing the computer processor to interpret a software test record using an identifier of a software test performed on the previous version of the memory software code associated with favorable software operation.

15. The dispersed storage network managing unit of claim 10 wherein the memory further includes instructions for causing the computer processor to load a second storage unit of the DSN with the first version of the memory software code and conduct at least a portion of the second test of the first version of the memory software code on the second storage unit.

16. The dispersed storage network managing unit of claim 10 wherein the second test is selected from a set of tests that includes the first test.

17. The dispersed storage network managing unit of claim 16 wherein the second test includes at least a portion of the first test.

18. The dispersed storage network managing unit of claim 16 wherein the second test is selected based, at least in part, on those areas of the memory software code that were changed since the previous version of the memory software code associated with favorable software operation.