Staging of write operations for container-based storage for sequential media

• US 10,216,424 B1
• Filed: 04/25/2018
• Issued: 02/26/2019
• Est. Priority Date: 04/25/2018
• Status: Active Grant

• Current Assignee: Western Digital Technologies Incorporated (Western Digital Corporation)
• Original Assignee: Western Digital Technologies Incorporated (Western Digital Corporation)
• Inventors: De Keyser, Koen, De Schrijver, Frederik Jacqueline Luc, Blyweert, Stijn, Christiaens, Mark
• Primary Examiner(s): Hindi, Nabil Z
• Application Number:
  US15/961,877
• Time in Patent Office: 307 Days
• Field of Search: 360 69
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0676 Magnetic disk device
• CPC Class Codes: G06F 3/0682 Tape device
• CPC Class Codes: G11B 20/1217 on discs
• CPC Class Codes: G11B 2020/1238 track, i.e. the entire a spirally or concentrically arranged path on which the recording marks are located

Abstract

A set of encoded data fragments is grouped into a container object in sequential order. Each encoded data fragment is a specific fragment size, and the container object is a specific container object size. The sequential order of the set of encoded data fragments can be tracked in a log in memory, such that the location of any one of the data fragments in the container object can be determined. The container object can be stored directly on a specific backend storage element, without using a file system. A corresponding container object identifier identifies the physical storage location of the container object on the backend storage element. The container object identifier is tracked in the log in memory, such that the physical location on the backend storage element of any specific one of the set of encoded data fragments in the container object can be determined.

Claims

1. A computer-implemented method, comprising:

encoding a data object into a plurality of encoded data fragments, each encoded data fragment being of a common fragment format and being a common data fragment size;

grouping a set of encoded data fragments into a container object in a sequential order, each encoded data fragment being the common data fragment size, and the container object being a specific predetermined container object size;

tracking, in a sequential transaction log, the sequential order of the set of encoded data fragments in the container object, wherein a location of any specific one of the set of encoded data fragments in the container object is a function of a position of the specific data fragment in the tracked sequential order and the common data fragment size;

transmitting the container object to a backend of a distributed encoded data storage system, wherein the container object is stored directly on a specific backend storage element;

in response to transmitting the container object, receiving a container object identifier corresponding to the container object, from the backend of the distributed encoded data storage system, wherein the container object identifier identifies a physical storage location of the container object on the specific backend storage element; and

tracking the received container object identifier in the sequential transaction log, wherein a physical location on the specific backend storage element of any specific one of the set of encoded data fragments in the container object is a function of a position of the specific data fragment in the tracked sequential order, the common data fragment size, and the physical storage location of the container object on the specific backend storage element as identified by the tracked container object identifier.

2. The computer-implemented method of claim 1, wherein:

the specific backend storage element further comprises a shingled magnetic recording (SMR) disk.

3. The computer-implemented method of claim 2, wherein:

the specific predetermined container object size corresponds to a size of a band of the SMR disk.

4. The computer-implemented method of claim 2, wherein storing the container object directly on the specific backend storage element further comprises:

storing the container object directly on a specific band of the SMR disk.

5. The computer-implemented method of claim 2, wherein a physical storage location of the container object on the specific backend storage element further comprises:

a specific band of the SMR disk.

6. The computer-implemented method of claim 1, further comprising:

transmitting a second container object containing a second set of encoded data fragments to the backend of the distributed encoded data storage system, wherein the second container object is stored sequentially after the first container object on the specific backend storage element;

in response to transmitting the second container object, receiving a second container object identifier corresponding to the second container object, from the backend of the distributed encoded data storage system, wherein the second container object identifier identifies a physical storage location of the second container object on the specific backend storage element; and

tracking the second received container object identifier in the sequential transaction log.

7. The computer-implemented method of claim 6, further comprising:

storing the first container object directly on a first specific band of a specific shingled magnetic recording (SMR) disk; and

storing the second container object directly on a second specific band of the specific SMR disk.

8. The computer-implemented method of claim 1, further comprising:

tracking data fragment-level write operations in the sequential transaction log;

reordering a random order sequence of tracked data fragment-level write operations while the tracked data fragment-level write operations are being processed in system memory, wherein reordering further comprises sorting the set of encoded data fragments being grouped into the container object into the tracked sequential order; and

subsequently to the reordering step, writing the container object from system memory to the specific backend storage element.

9. The computer-implemented method of claim 1, further comprising:

maintaining an association between the plurality of encoded data fragments and the data object.

10. The computer-implemented method of claim 1, further comprising:

receiving a read request from a client application targeting the data object;

retrieving the plurality of encoded data fragments from at least one container object on the specific backend storage element, using the sequential transaction log;

decoding the data object from the plurality of encoded data fragments; and

providing the data object to the client application.

11. The computer-implemented method of claim 1, further comprising:

retrieving a specific encoded data fragment from the specific backend storage element, using the sequential transaction log.

12. The computer-implemented method of claim 1, further comprising:

retrieving a specific container object from the specific backend storage element, using the sequential transaction log.

13. The computer-implemented method of claim 1, wherein:

the specific backend storage element further comprises a tape storage element.

14. The computer-implemented method of claim 1, wherein:

the specific backend storage element further comprises an optical storage element.

15. A computer system comprising:

a processor;

system memory;

a plurality of electromechanical backend storage elements;

instructions in the system memory programmed to encode a data object into a plurality of encoded data fragments, each encoded data fragment being of a common fragment format and being a common data fragment size;

instructions in the system memory programmed to group a set of encoded data fragments into a container object in a sequential order, each encoded data fragment being the common data fragment size, and the container object being a specific predetermined container object size;

instructions in the system memory programmed to track, in a sequential transaction log, the sequential order of the set of encoded data fragments in the container object, wherein a location of any specific one of the set of encoded data fragments in the container object is a function of a position of the specific data fragment in the tracked sequential order and the common data fragment size;

instructions in the system memory programmed to transmit the container object to a backend of a distributed encoded data storage system, wherein the container object is stored directly on a specific backend storage element without using a file system;

instructions in the system memory programmed to receive a container object identifier corresponding to the container object, from the backend of the distributed encoded data storage system, in response to transmitting the container object, wherein the container object identifier identifies a physical storage location of the container object on the specific backend storage element; and

instructions in the system memory programmed to track the received container object identifier in the sequential transaction log in system memory, wherein a physical location on the specific backend storage element of any specific one of the set of encoded data fragments in the container object is a function of a position of the specific data fragment in the tracked sequential order, the common data fragment size, and the physical storage location of the container object on the specific backend storage element as identified by the tracked object container object identifier.

16. The computer system of claim 15 wherein:

the specific backend storage element further comprises a shingled magnetic recording (SMR) disk.

17. The computer system of claim 16 wherein:

the specific predetermined container object size corresponds to a size of a band of the SMR disk.

18. The computer system of claim 16 further comprising:

instructions in the system memory programmed to store a first container object directly on a first specific band of the SMR disk; and

instructions in the system memory programmed to store the second container object directly on a second specific band of the specific SMR disk.

19. The computer system of claim 15 further comprising:

instructions in the system memory programmed to track data fragment-level write operations in the sequential transaction log in system memory;

instructions in the system memory programmed to reorder a random order sequence of tracked data fragment-level write operations while the tracked data fragment-level write operations are being processed in system memory, wherein reordering further comprises sorting the set of encoded data fragments being grouped into the container object into the tracked sequential order; and

instructions in the system memory programmed to write the container object from system memory to the specific backend storage element, subsequently to the reordering the random order sequence of tracked data fragment-level write operations.

20. A computer system comprising:

means for encoding a data object into a plurality of encoded data fragments, each encoded data fragment being of a common fragment format and being a common data fragment size;

means for grouping a set of encoded data fragments into a container object in a sequential order, each encoded data fragment being the common data fragment size, and the container object being a specific predetermined container object size;

means for tracking, in a sequential transaction log, the sequential order of the set of encoded data fragments in the container object, wherein a location of any specific one of the set of encoded data fragments in the container object is a function of a position of the specific data fragment in the tracked sequential order and the common data fragment size;

means for transmitting the container object to a backend of a distributed encoded data storage system, wherein the container object is stored directly on a specific backend storage element without using a file system;

means for receiving a container object identifier corresponding to the container object, from the backend of the distributed encoded data storage system, in response to transmitting the container object, wherein the container object identifier identifies a physical storage location of the container object on the specific backend storage element; and

means for tracking the received container object identifier in the sequential transaction log in system memory, wherein a physical location on the specific backend storage element of any specific one of the set of encoded data fragments in the container object is a function of a position of the specific data fragment in the tracked sequential order, the common data fragment size, and the physical storage location of the container object on the specific backend storage element as identified by the tracked container object identifier.