Method and apparatus for large scale data storage

• US 9,015,198 B2
• Filed: 05/24/2010
• Issued: 04/21/2015
• Est. Priority Date: 05/26/2009
• Status: Expired due to Fees

• Current Assignee: Pi-Coral Incorporated
• Original Assignee: Pi-Coral Incorporated
• Inventors: Osmond, Roger Frederick
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US12/785,993
• Publication Number:
  US 20100306269A1
• Time in Patent Office: 1,793 Days
• Field of Search: 707/609, 707637-640, 707692-700, 707790-793, 707821-825, 707827-831, 707/813, 707783-788, 711/4, 711/105, 711/111, 711113-114, 711208-209, 711162-163, 711/165, 711/112, 709/203, 709/206, 718/1
• US Class Current: 707/788
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/18 File system types
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 2216/03 Data mining
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A logical apparatus and associated methods provide highly scalable and flexible data storage in a network of computers. The apparatus provides flexible organizational and access control mechanisms and a practical and efficient way to work with smaller portions of a data storage system at a given time to enable sparse population, caching, paging and related functions. A data structure, called a virtual container, comprises references to objects stored in a data storage system such that the same object can be visible from different virtual containers, if such virtual containers hold references to said object. Access controls further enhance the effectiveness of the methods and structures to enable multiple simultaneous organizational schemes and selective sharing of objects. Participating nodes provide access to objects stored on said nodes and their participating peer nodes, employing the data storage apparatus, such that balance in the network is achieved by data placement decisions that may combine common constraints and a node'"'"'s individual self interest.

Claims

1. A virtual data storage system comprising:

at least one computing device operative to execute a virtual storage application, the virtual storage application configured to;

generate at least one virtual storage layer in operable communication with at least one logical storage layer configured to logically store a plurality of data objects within at least one logical storage container, the logical storage layer being in operable communication with at least one physical storage layer configured to physically store the plurality of data objects;

generate a plurality of virtual containers within the at least one virtual storage layer;

store at least one object reference in at least one of the plurality of virtual containers, the at least one object reference configured to reference at least one of the plurality of data objects; and

provide access to at least one of the plurality of data objects responsive to access of the at least one object reference, wherein access to the at least one object reference is based on an access policy of a virtual container storing the at least one object reference.

2. The virtual data storage system of claim 1, wherein the at least one object reference comprises one object reference configured to reference a plurality of data objects.

3. The virtual data storage system of claim 2, wherein the plurality of data objects comprises a plurality of elements associated with at least one multi-element object.

4. The virtual data storage system of claim 3, wherein the at least one multi-element object comprises a multimedia file and wherein the plurality of data objects comprise at least one of the following multimedia file elements:

an audio file, a video file, an image file, and a text file.

5. The virtual data storage system of claim 1, wherein the plurality of virtual containers comprises at least one container layer.

6. The virtual data storage system of claim 5, wherein the at least one container layer comprises a lowest layer and a plurality of higher layers and wherein the lowest layer comprises at least one cell comprising a hash table.

7. The virtual data storage system of claim 1, wherein the at least one computing device comprises a plurality of computing devices and the plurality of virtual containers are stored across the plurality of computing devices.

8. The virtual data storage system of claim 1, wherein the at least one reference is accessed using a uniform resource identifier.

9. A computer-implemented method for providing virtual data storage, the method comprising:

generating at least one virtual storage layer stored on at least one computing device, the at least one virtual storage layer being in operable communication with at least one logical storage layer configured to logically store a plurality of data objects within at least one logical storage container, the logical storage layer being in operable communication with at least one physical storage layer configured to physically store the plurality of data objects;

generating a plurality of virtual containers within the at least one virtual storage layer;

storing at least one object reference in at least one of the plurality of virtual containers;

referencing at least one of the plurality of data objects through the at least one object reference; and

providing access to at least one of the plurality of data objects responsive to access of the at least one object reference, wherein access to the at least one object reference is based on an access policy of a virtual container storing the at least one object reference.

10. The computer-implemented method claim 9, wherein the at least one computing device comprises a plurality of computing devices and arranging the plurality of virtual containers within the at least one virtual storage layer comprises storing the plurality of virtual containers across the plurality of computing devices.

11. The computer-implemented method of claim 9, wherein providing access to the plurality of data objects comprises providing access to the plurality of data objects using a uniform resource identifier.

12. The computer-implemented method of claim 9, further comprising obtaining information describing the plurality of data objects.

13. The computer-implemented method of claim 12, wherein the information describing the plurality of data objects comprises metadata.

14. The computer-implemented method of claim 12, wherein the information describing the plurality of data objects comprises a state of at least one of the plurality of data objects.

15. The computer-implemented method of claim 14, wherein the state comprises at least one of the following:

a data object is known to exist state, a data object is known to possibly exist state, a data object is not known to exist state, and a data object is known to exist state.

16. The computer-implemented method of claim 15, further comprising:

receiving, at the at least one computing device, a request to perform a search associated with at least one data object;

checking the state of the at least one data object; and

processing the request based on the state of the at least one data object.

17. The computer-implemented method of claim 16, wherein processing the search request comprises terminating the search responsive to the state of the at least one data object comprising the data object is not known to exist state.

18. The computer-implemented method of claim 16, wherein processing the search request comprises deferring the search responsive to the state of the at least one data object comprising the data object is known to possibly exist state.

19. The computer-implemented method of claim 16, wherein processing the search request comprises performing the search responsive to the state of the at least one data object comprising the data object is known to exist state.

20. The computer-implemented method of claim 9, further comprising controlling access to the plurality of virtual containers using a per-container access policy comprising at least one access condition and at least one access mode.