Scalable blob storage integrated with scalable structured storage

• US 8,620,884 B2
• Filed: 10/24/2008
• Issued: 12/31/2013
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Calder, Bradley Gene, Wang, Ju, Wu, Xinran, Nilakantan, Niranjan, Bhardwaj, Deepali, Srivastav, Shashwat, Nagy, Alexander Felsbki
• Primary Examiner(s): Kim, Paul
• Application Number:
  US12/257,747
• Publication Number:
  US 20100106695A1
• Time in Patent Office: 1,894 Days
• Field of Search: None
• US Class Current: 707/696
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement

Abstract

Embodiments of the present invention relate to systems, methods and computer storage media for facilitating the structured storage of binary large objects (Blobs) to be accessed by an application program being executed by a computing device. Generally, the structured storage of Blobs includes a primary structured storage index for indexing Blobs, a secondary hash index that is integrated into the structured storage system, a Blob log stream, and a Blob data stream for storing blocks that include the Blob data. In an embodiment, a block is created and written to a Blob store along with a block list. The block list facilitates the locating of one or more blocks that store the Blob data. In this embodiment, a primary structured storage index and a secondary hash index are updated to facilitate efficient access of the Blob in a structured storage system.

Claims

1. One or more computer storage media having a data structure embodied thereon for structured storage of one or more Binary Large Objects (Blobs) to be accessed by an application program being executed by a computing device, comprising:

a primary structured storage index indexing the one or more Blobs in a structured storage system;

a secondary hash index maintaining a compressed hash index of the primary structured storage index, the secondary hash index facilitates efficient lookup of a Blob primary key, wherein the secondary hash index includes one or more hash values representing the one or more primary keys associated with the one or more Blobs; and

a Blob store storing the one or more Blobs, wherein the Blob store is mapped to the primary structured storage index using a Blob log stream and the secondary hash index using a Blob data stream.

2. The computer storage media of claim 1, wherein the Blob store comprises the Blob log stream that includes one or more uncommitted blocks for storing the one or more Blobs.

3. The computer storage media of claim 2, wherein the Blob log stream includes a record for updates made to the primary structured storage index.

4. The computer storage media of claim 3, wherein the one or more blocks and the record are written to the Blob log stream in a single write.

5. The computer storage media of claim 3, wherein a table server that loads a partition that includes at least one of the one or more Blobs, reconstructs a memory table for the table server based, in part, on the Blob log stream.

6. The computer storage media of claim 1, wherein the primary structured storage index includes at least one Blob property column.

7. The computer storage media of claim 6, wherein at least one Blob property includes information corresponding to at least one selection from the following,

a pointer to the Blob associated with the Blob property,

a pointer to a Temporary Block List (TBL), wherein the TBL maintains a list of one or more uncommitted blocks corresponding to the Blob associated with the Blob property,

a pointer to a Committed Block List (CBL), wherein the CBL maintains a list of one or more committed blocks corresponding to the Blob associated with the Blob property,

a last modified time for the TBL,

a size of the Blob associated with the Blob property,

a creation time of the Blob associated with the Blob property, and

metadata of the Blob associated with the Blob property.

8. The computer storage media of claim 1, wherein the secondary hash index is eagerly updated after a memory table of a partition served from a table server is updated.

9. The computer storage media of claim 1, wherein the secondary hash index is lazily updated after a memory table of a partition served from a table server is written to a file table of the partition.

10. The computer storage media of claim 9, wherein a queue associated with the secondary hash index includes an indication of additional updates pending for the secondary hash index, wherein the secondary hash index is useable when the queue is empty.

11. The computer storage media of claim 1, wherein the secondary hash index facilitates a single in-memory lookup of one of the one or more Blobs.

12. A computer-implemented system utilizing a processor and memory for structured storage of one or more Binary Large Objects (Blobs) to be accessed by an application program being executed by a computing device, comprising:

a primary structured storage index indexing the one or more Blobs in a structured storage system;

a secondary hash index maintaining a compressed hash index of the primary structured storage, the secondary hash index facilitates efficient lookup of a Blob primary key, wherein the secondary hash index includes one or more hash values representing the one or more primary keys associated with the one or more Blobs; and

a Blob store storing the one or more Blobs, wherein the Blob store comprises a Blob log stream having one or more uncommitted blocks and a Blob data stream having one or more committed blocks, wherein the primary structured storage index is indirectly mapped to the Blob data stream via the Blob log stream and the secondary hash index is directly mapped to the Blob data stream.

13. The computer-implemented system of claim 12, wherein the Blob log stream is mapped to the Blog data stream using a pointer to a Temporary Block List (TBL), wherein the TBL maintains a list of the one or more uncommitted blocks corresponding to a Blob associated with the Blob property.

14. The computer-implemented system of claim 13, wherein the Blob log stream includes a record for updates made to the primary structured storage index.

15. The computer-implemented system of claim 14, wherein a table server that loads a partition that includes at least one of the one or more Blobs, reconstructs a memory table for the table server based, in part, on the Blob log stream.

16. The computer-implemented system of claim 15, wherein at least one Blob property includes information corresponding to at least one selection from the following,

a pointer to a Blob associated with the Blob property,

a pointer to a Temporary Block List (TBL), wherein the TBL maintains a list of one or more uncommitted blocks corresponding to the Blob associated with the Blob property,

a pointer to a Committed Block List (CBL), wherein the CBL maintains a list of one or more committed blocks corresponding to the Blob associated with the Blob property,

a last modified time for the TBL,

a size of the Blob associated with the Blob property,

a creation time of the Blob associated with the Blob property, and

metadata of the Blob associated with the Blob property.

17. The computer-implemented system of claim 12, wherein the secondary hash index is eagerly updated after a memory table of a partition served from a table server is updated.

18. The computer-implemented system of claim 12, wherein the secondary hash index is lazily updated after a memory table of a partition served from a table server is written to a file table of the partition.

19. The computer-implemented system of claim 18, wherein a queue associated with the secondary hash index includes an indication of additional updates pending for the secondary hash index, wherein the secondary hash index is useable when the queue is empty.

20. The computer-implemented system of claim 12, wherein the secondary hash index facilitates a single in-memory lookup of one of the one or more Blobs.

21. One or more computer storage media having a data structure embodied thereon for structured storage of one or more Binary Large Objects (Blobs) to be accessed by an application program being executed by a computing device, comprising:

a primary structured storage index indexing the one or more Blobs in a structured storage system;

a secondary hash index maintaining a compressed in-memory hash index of the primary structure storage index, the secondary hash index is an in-memory hash index that facilitates efficient lookup of a Blob primary key, wherein the secondary hash index includes one or more hash values representing the one or more primary keys associated with the one or more Blobs;

a Blob store storing the one or more Blobs wherein the Blob store comprises a Blob log stream and a Blob data stream, wherein the Blob log stream processes single input/output operations in association with the primary structure storage index and the Blob data stream processes single in-memory lookups in association with the secondary hash index; and

a table scheme that defines one or more data items included in the secondary hash index and a Blob header of the Blob data stream, wherein the one or more data items comprise Blob meta data that are retrieved when an associated Blob of the Blob meta data is accessed.