Import, export, and copy management for tiered object storage

• US 10,642,809 B2
• Filed: 06/26/2017
• Issued: 05/05/2020
• Est. Priority Date: 06/26/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bhagwat, Deepavali M., Bhosale, Nilesh P., Dain, Joseph W., Hewitt, James, Sawdon, Wayne A.
• Primary Examiner(s): Corrielus, Jean M
• Application Number:
  US15/633,599
• Publication Number:
  US 20180373745A1
• Time in Patent Office: 1,044 Days
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]

Abstract

In one embodiment, a method includes receiving, at a centralized index operating in a cloud environment via a messaging bus, first events spawned by a first processor managing an object store accessible via the cloud environment. The method also includes receiving, at the centralized index, second events spawned by a second processor managing an archival store accessible via the cloud environment. Also, the method includes normalizing metadata in the first and second events. Moreover, the method includes indexing and storing the metadata in the centralized index. The object store and the archival store operate in different object subsystems of the cloud environment. Other systems, methods, and computer program products are described for managing import, export, and copy of data within a public tiered object storage environment in accordance with more embodiments.

Claims

1. A method, comprising:

receiving, via a messaging bus at a centralized index operating in a cloud environment, a first event indicating an action performed on first data within an object store accessible via the cloud environment;

receiving, at the centralized index, a second event indicating an action performed on second data within an archival store separate from the object store that is accessible via the cloud environment, where the object store and the archival store operate in different object subsystems of the cloud environment;

normalizing metadata in the first event and the second event to correspond to a single namespace across the different object subsystems; and

indexing and storing the normalized metadata in the centralized index, where the normalized metadata includes the action performed on the first data, a storage location of the first data within the object store, the action performed on the second data, and a storage location of the second data within the archival store.

2. The method as recited in claim 1, wherein the object store comprises a plurality of non-volatile memory storage devices, wherein the archival store comprises a plurality of higher latency media, and wherein the object store and the archival store operate in a heterogeneous storage environment.

3. The method as recited in claim 1, wherein the object store and the archival store operate according to different protocols and utilize different data structures and data architectures.

4. The method as recited in claim 1, further comprising:

receiving a request to move the first data from the object store to the archival store;

determining the storage location of the first data within the object store, utilizing the centralized index;

responding to the request with the storage location of the first data within the object store;

receiving, at the centralized index, a third event indicating a storage of the first data within the archival store;

setting a timer for maintaining the first data within the object store; and

deleting the first data within the object store, in response to an expiration of the timer.

5. The method as recited in claim 1, further comprising:

receiving a search request for third data;

searching the centralized index for metadata relating to a storage location of the third data; and

responding to the search request with the storage location for the third data,

wherein the storage location for the third data is reported on the object store in response to a determination that at least one copy of the third data exists on the object store, and wherein the storage location for the third data is reported on the archival store in response to a determination that at least one copy of the third data exists on the archival store and not on the object store.

6. The method as recited in claim 1, further comprising:

receiving a request to duplicate the second data within the archival store;

determining the storage location of the second data within the archival store, utilizing the centralized index;

responding to the request with the storage location of the second data within the archival store;

receiving, at the centralized index, a third event indicating a copying of the second data to an additional storage location within the object store;

adding, to the normalized metadata in the centralized index, the additional storage location of the second data within the object store to the storage location of the second data within the archival store;

receiving a search request for the second data;

searching the centralized index for the normalized metadata relating to a storage location of the second data; and

responding to the search request with the additional storage location within the object store, in response to determining that at least one copy of the second data exists within the object store.

7. The method as recited in claim 1, further comprising:

receiving a request to move the first data from the object store to the archival store;

determining the storage location of the first data within the object store, utilizing the centralized index;

responding to the request with the storage location of the first data within the object store;

receiving, at the centralized index, a third event indicating a storage of the first data within the archival store;

setting a timer for maintaining the first data within the object store;

deleting the first data within the object store, in response to an expiration of the timer;

receiving a request to duplicate the second data within the archival store;

determining the storage location of the second data within the archival store, utilizing the centralized index;

responding to the request with the storage location of the second data within the archival store;

receiving, at the centralized index, a third event indicating a copying of the second data to an additional storage location within the object store;

adding, to the normalized metadata in the centralized index, the additional storage location of the second data within the object store to the storage location of the second data within the archival store;

receiving a search request for the second data;

searching the centralized index for the normalized metadata relating to a storage location of the second data; and

responding to the search request with the additional storage location within the object store, in response to determining that at least one copy of the second data exists within the object store.

8. A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the embodied program instructions being executable by a processing circuit to cause the processing circuit to:

receive, by the processing circuit via a messaging bus at a centralized index operating in a cloud environment, a first event indicating an action performed on first data within an object store accessible via the cloud environment;

receive, by the processing circuit at the centralized index, a second event indicating an action performed on second data within an archival store separate from the object store that is accessible via the cloud environment, where the object store and the archival store operate in different object subsystems of the cloud environment;

normalize, by the processing circuit, metadata in the first event and the second event to correspond to a single namespace across the different object subsystems; and

index and store, by the processing circuit, the normalized metadata in the centralized index, where the normalized metadata includes the action performed on the first data, a storage location of the first data within the object store, the action performed on the second data, and a storage location of the second data within the archival store.

9. The computer program product as recited in claim 8, wherein the object store comprises a plurality of non-volatile memory storage devices, wherein the archival store comprises higher latency media, and wherein the object store and the archival store operate in a heterogeneous storage environment.

10. The computer program product as recited in claim 8, wherein the object store and the archival store operate according to different protocols and utilize different data structures and data architectures.

11. The computer program product as recited in claim 8, wherein the embodied program instructions are further executable by the processing circuit to cause the processing circuit to:

receive, by the processing circuit, a search request for third data;

search, by the processing circuit, the centralized index for metadata relating to a storage location of the third data; and

respond, by the processing circuit, to the search request with the storage location for the third data,

wherein the storage location for the third data is reported on the object store in response to a determination that at least one copy of the third data exists on the object store, and wherein the storage location for the third data is reported on the archival store in response to a determination that at least one copy of the third data exists on the archival store and not on the object store.

12. The computer program product as recited in claim 8, wherein the embodied program instructions are further executable by the processing circuit to cause the processing circuit to:

receive, by the processing circuit, a request to move the first data from the object store to the archival store;

determine, by the processing circuit, the storage location of the first data within the object store, utilizing the centralized index;

respond, by the processing circuit, to the request with the storage location of the first data within the object store;

receive, by the processing circuit at the centralized index, a third event indicating a storage of the first data within the archival store;

set, by the processing circuit, a timer for maintaining the first data within the object store; and

delete, by the processing circuit, the first data within the object store, in response to an expiration of the timer.

13. The computer program product as recited in claim 8, wherein the embodied program instructions are further executable by the processing circuit to cause the processing circuit to:

receive, by the processing circuit, a request to duplicate the second data within the archival store;

determine, by the processing circuit, the storage location of the second data within the archival store, utilizing the centralized index;

respond, by the processing circuit, to the request with the storage location of the second data within the archival store;

receive, by the processing circuit at the centralized index, a third event indicating a copying of the second data to an additional storage location within the object store;

add, by the processing circuit to the normalized metadata in the centralized index, the additional storage location of the second data within the object store to the storage location of the second data within the archival store;

receive, by the processing circuit, a search request for the second data;

search, by the processing circuit, the centralized index for the normalized metadata relating to a storage location of the second data; and

respond, by the processing circuit, to the search request with the additional storage location within the object store, in response to determining that at least one copy of the second data exists within the object store.

14. The computer program product as recited in claim 8, wherein the embodied program instructions are further executable by the processing circuit to cause the processing circuit to:

receive, by the processing circuit, a request to move the first data from the object store to the archival store;

determine, by the processing circuit, the storage location of the first data within the object store, utilizing the centralized index;

respond, by the processing circuit, to the request with the storage location of the first data within the object store;

receive, by the processing circuit at the centralized index, a third event indicating a storage of the first data within the archival store;

set, by the processing circuit, a timer for maintaining the first data within the object store;

delete, by the processing circuit, the first data within the object store, in response to an expiration of the timer;

receive, by the processing circuit, a request to duplicate the second data within the archival store;

determine, by the processing circuit, the storage location of the second data within the archival store, utilizing the centralized index;

respond, by the processing circuit, to the request with the storage location of the second data within the archival store;

receive, by the processing circuit at the centralized index, a third event indicating a copying of the second data to an additional storage location within the object store;

add, by the processing circuit to the normalized metadata in the centralized index, the additional storage location of the second data within the object store to the storage location of the second data within the archival store;

receive, by the processing circuit, a search request for the second data;

search, by the processing circuit, the centralized index for the normalized metadata relating to a storage location of the second data; and

respond, by the processing circuit, to the search request with the additional storage location within the object store, in response to determining that at least one copy of the second data exists within the object store.

15. A system, comprising:

a processing circuit;

a memory; and

logic stored to the memory, that when executed by the processing circuit causes the processing circuit to;

receive, by the processing circuit via a messaging bus at a centralized index operating in a cloud environment, a first event indicating an action performed on first data within an object store accessible via the cloud environment;

receive, by the processing circuit at the centralized index, a second event indicating an action performed on second data within an archival store separate from the object store that is accessible via the cloud environment, where the object store and the archival store operate in different object subsystems of the cloud environment;

normalize, by the processing circuit, metadata in the first event and the second event to correspond to a single namespace across the different object subsystems; and

index and store, by the processing circuit, the normalized metadata in the centralized index, where the normalized metadata includes the action performed on the first data, a storage location of the first data within the object store, the action performed on the second data, and a storage location of the second data within the archival store.