Blob manipulation in an integrated structured storage system

• US 8,495,036 B2
• Filed: 10/24/2008
• Issued: 07/23/2013
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Calder, Bradley Gene, Wang, Ju, Wu, Xinran, Nilakantan, Niranjan, Srivastav, Shashwat, Bhardwaj, Deepali, Nagy, Alexander Felsbki
• Primary Examiner(s): NGUYEN, PHONG H
• Application Number:
  US12/257,786
• Publication Number:
  US 20100106734A1
• Time in Patent Office: 1,733 Days
• Field of Search: 707/696, 707/758, 707/791
• US Class Current: 707/696
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

Embodiments of the present invention relate to systems, methods and computer storage media for facilitating the structured storage of binary large objects (Blobs) to be accessed by an application program being executed by a computing device. Generally, the manipulation of Blobs in a structured storage system includes receiving a request for a Blob, which may be located by way of a Blob pointer. The Blob pointer allows for the data, such as properties, of the Blob to be identified and located. Expired properties are garbage collected as a manipulation of the Blob data within a structured storage system. In an embodiment, the Blob is identified by a key that is utilized within a primary structured index to located the requested Blob. In another embodiment, the requested Blob is located utilizing a secondary hash index. In an additional embodiment, the Blob is locate utilizing a file table.

Claims

1. One or more tangible computer storage media having computer-executable instructions embodied thereon for performing a method for retrieving data of a Binary Large Object (Blob) in a structured storage system, the method comprising:

receiving a request for a Blob from an application;

prior to searching a persistent file table of the structured storage system for a Blob pointer associated with the Blob, identifying within an in-memory hash index (“

hash index”

) a Blob pointer associated with the Blob, the hash index is a compressed hash index of the file table, wherein the structured storage system comprises a sequence number allocator, wherein the sequence number allocator generates sequence numbers that are assigned to structured storage system; and

further wherein the sequence number allocator maintains an indication of a last acknowledged sequence, which is the highest sequence number completed, further wherein, during replay of a Blob log stream, transaction with a sequence number less than the last acknowledged sequence are allowed to replay from the Blob log stream;

identifying, with a processor, data associated with the Blob, wherein the data is identified, in part, by the Blob pointer; and

communicating the data to the application.

2. The method of claim 1, wherein identifying data further comprising:

identifying a Committed Block List (CBL) from the Blob pointer;

identifying one or more blocks of a Blob stream, wherein the one or more blocks are identified by the CBL; and

retrieving data from at least one of the one or more blocks.

3. The method of claim 1, wherein identifying the Blob pointer further comprises searching at least one of a memory table, the hash index, and a file table of the structured storage system, wherein the memory table is an in-memory version of updates for an update log of a partition.

4. The method of claim 3, wherein the memory table is searched first and the hash index is searched second until the Blob pointer is identified.

5. The method of claim 4, wherein the hash index is searched when a key is not found in the memory table.

6. The method of claim 1, wherein identifying the Blob pointer further comprises:

determining a key for the Blob is unavailable in a memory table;

and identifying the Blob pointer for the Blob in the hash index.

7. The method of claim 1, wherein identifying data of the Blob further comprises:

identifying a key in the hash index;

identifying that one or more dirty bits are not associated with the Blob; and

retrieving the data from a Blob header of the Blob.

8. The method of claim 1, further comprising

identifying a key associated with the Blob pointer, wherein the key includes a reference to a particular row of an index;

locating the Blob pointer in the hash index, wherein the Blob pointer is located utilizing the key;

utilizing the Blob pointer to identify a Committed Block List (CBL) that identifies one or more blocks of a Blob data stream; and

reading data from at least one of the one or more blocks identified in the CBL.

9. A structured storage system having a processor and memory that facilitates the seamless integration of one or more Binary Large Objects (Blobs), the structured storage system comprising:

a processor;

computer readable media;

a primary structured index embodied on the computer readable media, the primary structured index indexes one or more Blobs in a structured storage system, the primary structured storage index has at least a memory table and a row store, wherein the row store includes at least an update log stream and a file checkpoint stream, wherein the structured storage system comprises a sequence number allocator, wherein the sequence number allocator generates sequence numbers that are assigned to structured storage system; and

further wherein the sequence number allocator maintains an indication of a last acknowledged sequence, which is the highest sequence number completed, further wherein, during replay of a Blob log stream, transaction with a sequence number less than the last acknowledged sequence are allowed to replay from the Blob log stream; and

a secondary hash index embodied on the computer readable media, the secondary hash index is a compressed hash index of the primary structured index, wherein the secondary hash index facilitates in-memory lookup of one or more Blobs indexed in the primary structured index prior to searching the primary structured index for the one or more Blobs.

10. The structured storage system of claim 9, wherein the memory table is checkpointed in preparation of an offload of an associated partition.

11. The structured storage system of claim 10, wherein the secondary hash index is checkpointed in preparation of an offload of an associated partition.

12. The structured storage system of claim 9, wherein the secondary hash index is maintained consistent with the primary structured index by loading of the secondary hash index from a last checkpoint and utilizing at least one of:

1) a file table and the memory table, and

2) a hash index delta log and memory table changes.

13. The structured storage system of claim 9 further comprising a Blob store, wherein the Blob store allows for multiple Blob properties per row of the Blob store, further wherein a key is associated with each Blob property of the Blob store, the key is comprised of a primary row key associated with the row of the Blob property and a Blob property name associated with the Blob property.