Log consolidation

• US 9,063,980 B2
• Filed: 12/21/2012
• Issued: 06/23/2015
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Dhamankar, Robin Dhananjay, Byrne, Peter, Kodavalla, Hanumantha Rao
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Pham, Michael
• Application Number:
  US13/723,961
• Publication Number:
  US 20140181036A1
• Time in Patent Office: 914 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/1805 Append-only file systems, e.g. using logs or journals to store data
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

Consolidating a set of tenant log streams from separate user databases into a consolidated log stream. The method includes receiving a plurality of tenant log streams from separate user data bases. The method further includes recording the plurality of tenant log streams as a consolidated log stream. The method further includes maintaining metadata about the consolidated log stream to map log records from the plurality of tenant log streams to their location in the consolidated log stream.

Claims

1. In a virtual computing environment that includes a plurality of tenant databases, a computer-implemented method of consolidating at a consolidated log host a set of tenant log streams from the plurality of tenant databases into a consolidated log stream so that the number of individual tenant log streams consolidated at the consolidated log host are independent of whether a number of disks are available at the consolidated log host, the computer-implemented method comprising:

at a virtualization layer of a log consolidation host, intercepting I/O requests from the plurality of tenant log streams from the plurality of tenant databases;

linearizing the intercepted I/O requests from the plurality of tenant log streams into a physical log file that forms a consolidated log stream;

storing the consolidated log stream in a log file;

generating a map from metadata associated with each tenant log stream, the map identifying log records from the plurality of tenant log streams, and the log records being mapped to corresponding locations in the consolidated log stream to provide correlation of data associated with each tenant log stream consolidated into the consolidated log stream with a tenant database from which said data originated; and

storing the map.

2. The computer-implemented method of claim 1, wherein generating the map from the metadata comprises updating metadata in mapping tables of the map as I/O requests for one or more of the plurality of tenant databases processed using the consolidated log stream.

3. The computer-implemented method of claim 1, wherein generating the map comprises mapping a stream identifier and an offset in each tenant log stream to a corresponding offset in the consolidated log stream.

4. The computer-implemented method of claim 1, generating the map comprises mapping a stream identifier and a sector number for each tenant log stream to a log sequence number of a corresponding tenant log record that has data for the sector number.

5. The computer-implemented method of claim 1, further comprising exposing file system like APIs over the consolidated log stream to a database engine.

6. The computer-implemented method of claim 1, wherein storing the map comprises using native indexing methods of a backing database to perform efficient and persistent mapping using the consolidated log stream.

7. The computer-implemented method of claim 1, further comprising recovering one or more of the plurality of tenant databases by first recovering a database for the consolidated log stream and then recovering tenant log streams for individual tenant databases.

19. The system architecture of claim 3, wherein one or more of the tenant databases are recovered by first recovering a database for the consolidated log stream and then recovering tenant log streams for individual tenant databases.

8. In a virtual computing environment that includes a plurality of tenant databases, a computer readable storage device comprising computer executable instructions that when executed by one or more processors cause one or more processors to perform a computer-implemented method of consolidating a set of tenant log streams from the plurality of tenant databases into a consolidated log stream so that individual tenant log streams do not need to be individually stored on separate disks corresponding to each tenant log stream, the computer-implemented method comprising:

at a virtualization layer of a log consolidation host, intercepting 110 requests from the plurality of tenant log streams from the plurality of tenant databases;

linearizing the intercepted I/O requests from the plurality of tenant log streams into a physical log file that forms a consolidated log stream;

storing the consolidated log stream in a log file;

generating a map from metadata associated with each tenant log stream, the map identifying log records from the plurality of tenant log streams, the log records being mapped to corresponding locations in the consolidated log stream to provide correlation of data associated with each tenant log stream consolidated into the consolidated log stream with a tenant database from which said data originated; and

storing the map.

9. The computer readable storage device of claim 8, wherein generating the map from the metadata comprises updating metadata in mapping tables of the map as I/O requests for one or more of the plurality of tenant databases processed using the consolidated stream.

10. The computer readable storage device of claim 8, wherein generating the map comprises mapping a stream identifier and an offset in each tenant log stream to a corresponding offset in the consolidated log stream.

11. The computer readable storage device of claim 8, generating the map comprises mapping a stream identifier and a sector number for each tenant log stream to a log sequence number of a corresponding tenant log record that has data for the sector number.

12. The computer readable storage device of claim 8, further comprising exposing file system like APIs over the consolidated log stream to a database engine.

13. The computer readable storage device of claim 8, wherein storing the map comprises using native indexing methods of a backing database to perform efficient and persistent mapping using the consolidated log stream.

14. The computer readable storage device of claim 8, further comprising recovering one or more of the plurality of tenant databases by first recovering a database for the consolidated log stream and then recovering tenant log streams for individual tenant databases.

15. In a virtual computing environment that includes a plurality of tenant databases, a system architecture for a consolidated host that consolidates a set of tenant log streams from a plurality of tenant databases into a consolidated log stream so that individual tenant log streams do not need to be individually stored on separate disks corresponding to each tenant log stream, the system architecture for the consolidated host comprising:

a virtualization layer that intercepts 110 requests from a plurality of tenant log streams from a plurality of tenant databases;

a log file in which the consolidated log stream is stored;

one or more map tables that store a map from metadata associated with each tenant log stream;

one or more scanners which scan intercepted I/O requests from one or more tenant databases;

a log manager that processes intercepted and scanned I/O requests from one or more tenant databases;

one or more processors; and

one or more storage memories storing executable instructions, which when executed by the one or more processors, causes the following computer-implemented method to be performed;

intercepting at said virtualization layer I/O requests from the plurality of tenant log streams from the plurality of tenant databases;

linearizing the intercepted I/O requests from the plurality of tenant log streams into a physical log file that forms a consolidated log stream;

storing the consolidated log stream in the log file;

generating a map from metadata associated with each tenant log stream, the map identifying log records from the plurality of tenant log streams, and the log records being mapped to corresponding locations in the consolidated log stream to provide correlation of data associated with each tenant log stream consolidated into the consolidated log stream with a tenant database from which said data originated; and

storing the map in the one or more map tables.

16. The system architecture of claim 15, wherein the stored map comprises updated metadata that is stored in mapping tables as I/O requests for one or more tenant databases are processed using the consolidated log stream.

17. The system architecture of claim 15, wherein the stored map comprises a stream identifier and an offset in each tenant log stream that maps to a corresponding offset in the consolidated log stream.

18. The system architecture of claim 15, wherein the stored map comprises a stream identifier and a sector number for each tenant log stream that are mapped to a log sequence number of a corresponding tenant log record that has data for the sector number.

20. The system architecture of claim 15, wherein the map is stored using native indexing methods of a backing database to perform efficient and persistent mapping.