LOG CONSOLIDATION

• US 20140181036A1
• Filed: 12/21/2012
• Published: 06/26/2014
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kodavalla, Hanumantha Rao, Dhamankar, Robin Dhananjay, Byrne, Peter
• Granted Patent: US 9,063,980 B2
• Time in Patent Office: Days
• US Class Current: 707/648
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/1805 Append-only file systems, e.g. using logs or journals to store data
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

Consolidating a set of tenant log streams from separate user databases into a consolidated log stream. The method includes receiving a plurality of tenant log streams from separate user data bases. The method further includes recording the plurality of tenant log streams as a consolidated log stream. The method further includes maintaining metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

Claims

1. A method of consolidating a set of tenant log streams from separate user databases into a consolidated log stream, the method comprising:

receiving a plurality of tenant log streams from separate user data bases;

recording the plurality of tenant log streams as a consolidated log stream; and

maintaining metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

2. The method of claim 1, wherein maintaining metadata comprises updating metadata in mapping tables to identify where data in the consolidated log stream came from.

3. The method of claim 1, wherein recording the plurality of physical log streams as a consolidated log stream is performed by virtualizing the plurality of physical log streams into the consolidated log stream.

4. The method of claim 3, further comprising providing asynchronous I/O abstractions to the tenant log files to facilitate the virtualization.

5. The method of claim 3, wherein virtualizing the plurality of physical log streams into the consolidated log stream comprises exposing file system like APIs over the consolidated log stream to a database engine.

6. The method of claim 1, wherein maintaining metadata comprises using native indexing methods of a backing database to perform efficient and persistent mapping using the consolidated log stream.

7. The method of claim 1, further comprising recovering one or more of the separate user databases, wherein recovering one or more of the separate user databases comprises first recovering the database for the consolidated log stream and then recovering log streams for individual tenants.

19. The method of claim 3, wherein virtualizing the plurality of physical log streams into the consolidated log stream comprises exposing file system like APIs over the consolidated log stream to a database engine.

8. A computer readable storage medium comprising computer executable instructions that when executed by one or more processors cause one or more processors to perform the following:

receiving a plurality of tenant log streams from separate user data bases;

recording the plurality of tenant log streams as a consolidated log stream; and

maintaining metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

9. The computer readable storage medium of claim 8, wherein maintaining metadata comprises updating metadata in mapping tables to identify where data in the consolidated log stream came from.

10. The computer readable storage medium of claim 8, wherein recording the plurality of physical log streams as a consolidated log stream is performed by virtualizing the plurality of physical log streams into the consolidated log stream.

11. The computer readable storage medium of claim 10, further comprising providing asynchronous I/O abstractions to the tenant log files to facilitate the virtualization.

12. The computer readable storage medium of claim 10, wherein virtualizing the plurality of physical log streams into the consolidated log stream comprises exposing file system like APIs over the consolidated log stream to a database engine.

13. The computer readable storage medium of claim 8, wherein maintaining metadata comprises using native indexing methods of a backing database to perform efficient and persistent mapping.

14. The computer readable storage medium of claim 8, further comprising recovering one or more of the separate user databases, wherein recovering one or more of the separate user databases comprises first recovering the database for the consolidated log stream and then recovering log streams for individual tenants.

15. A system for consolidating a set of tenant log streams from separate user databases into a consolidated log stream, the system comprising:

a consolidated log, wherein the consolidated log is configured to;

receive a plurality of tenant log streams from file I/O mechanisms of separate user data bases;

record the plurality of tenant log streams as a consolidated log stream; and

update metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

16. The system of claim 15, wherein maintaining metadata comprises updating metadata in mapping tables to identify where data in the consolidated log stream came from.

17. The system of claim 15, wherein recording the plurality of physical log streams as a consolidated log stream is performed by virtualizing the plurality of physical log streams into the consolidated log stream.

18. The system of claim 17, wherein the system is comprised of standard database components, the system further comprising a thin virtualization layer to intercept I/O requests and serve them to the consolidated log.

20. The system of claim 15, wherein maintaining metadata comprises using native indexing methods of a backing database to perform efficient and persistent mapping.