Object store management operations within compute-centric object stores

• US 9,792,290 B2
• Filed: 03/25/2015
• Issued: 10/17/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Joyent, Inc. (Samsung Electronics Co. Ltd.)
• Original Assignee: Joyent, Inc. (Samsung Electronics Co. Ltd.)
• Inventors: Cavage, Mark, Fitch, Nathan, Kuo, Fred, Xiao, Yunong, Pacheco, David, Cantrill, Bryan
• Primary Examiner(s): Hashemi, Sana Al
• Application Number:
  US14/668,425
• Publication Number:
  US 20150199372A1
• Time in Patent Office: 937 Days
• Field of Search: 707607, 707661, 707791, 707820, 707828
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/289 Object oriented databases

Abstract

Object store management operations within compute-centric object stores are provided herein. An exemplary method may include transforming an object storage dump into an object store table by a table generator container, wherein the object storage dump includes at least objects within an object store that are marked for deletion, transmitting records for objects from the object store table to reducer containers, such that each reducer container receives object records for at least one object, the object records comprising all object records for the at least one object, generating a set of cleanup tasks by the reducer containers, and executing the cleanup tasks by a cleanup agents.

Claims

1. A method, comprising:

transforming an object storage dump into an object store table by a table generator container, wherein the object storage dump includes at least objects within an object store that are marked for deletion;

transmitting records for objects from the object store table to reducer containers, such that each reducer container receives object records for at least one object, the object records comprising all object records for the at least one object;

generating a set of cleanup tasks by the reducer containers;

storing the cleanup tasks in a directory; and

executing the cleanup tasks by a cleanup agents, wherein the cleanup agents periodically look in the directory for the cleanup tasks.

4. The method according to claim 1, wherein objects marked for deletion are indexed in a log record that is stored on the object store.

5. The method according to claim 1, wherein a first field in each record to the object store table comprises an object identifier and a second field of the record comprises a time that the record was generated.

6. The method according to claim 1, wherein generating a set of cleanup tasks by the reducer containers comprises each reducer sorting the records for an object such that the objects are placed in ranked order according to time, evaluating the records in chronological order, and determining records that include objects that are collected as garbage.

7. The method according to claim 1, wherein records for an object are collected as garbage if the records for the object referenced in a log record have one or more later records that reference the object.

8. The method according to claim 1, wherein a cleanup agent is configured to download a cleanup task, locate objects referenced in the cleanup task, and delete the objects from the object store if the cleanup agent determines that the objects are deleteable.

11. The method according to claim 6, wherein a record includes an object that is collected as garbage only if the record has passed a grace period established for the object store.

12. The method according to claim 8, wherein the cleanup agent updates shards that comprise metadata that comprise any of ownership, size, free space, and combinations thereof, relative to a namespace that comprises the objects.

2. A system, comprising:

one or more processors; and

logic encoded in one or more tangible media for execution by the one or more processors, the logic comprising;

a table generator container that transforms an object storage dump into an object store table, wherein the object storage dump includes at least objects within an object store that are marked for deletion;

a set reducer containers that;

receive records for objects from the object store table from the table generator, such that each reducer container receives object records for at least one object, the object records comprising all object records for the at least one object;

sort the records for an object such that the object records are placed in ranked order according to time, evaluating the object records in chronological order, and determining object records that include objects that are collected as garbage, wherein records for an object are collected as garbage if the records for the object referenced in a log record have one or more later records that reference the object;

generate a set of cleanup tasks; and

storing;

the cleanup tasks in a directory; and

cleanup agents that execute the cleanup tasks, wherein the cleanup agents periodically look in the directory for the cleanup tasks.

9. The system according to claim 2, wherein a first field in each record to the object store table comprises an object identifier and a second field of the record comprises a time that the record was generated.

10. The system according to claim 2, wherein the reducer containers generate a set of cleanup tasks in such a way that each reducer:

walks the records in chronological order to determine records that are to be deleted.

3. A method, comprising:

transforming an object storage dump into an object store table by a table generator container, wherein the object storage dump includes at least objects within an object store that are marked for deletion;

transmitting records for objects from the object store table to reducer containers, such that each reducer container receives object records for at least one object, the object records comprising all object records for the at least one object;

generating a set of cleanup tasks by the reducer containers by;

each reducer sorting the records for an object such that the object records are placed in ranked order according to time, evaluating the object records in chronological order, and determining object records that include objects that are collected as garbage, wherein the object records include an object that is collected as garbage only if the record has passed a grace period established for the object store;

storing the cleanup tasks in a directory; and

executing the cleanup tasks by a cleanup agents, wherein the cleanup agents periodically look in the directory for the cleanup tasks.