Apparatus and Method for Read Optimized Bulk Data Storage

• US 20110010330A1
• Filed: 07/08/2009
• Published: 01/13/2011
• Est. Priority Date: 07/08/2009
• Status: Active Grant

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Greenplum, Inc. (Dell Technologies Inc.)
• Inventors: Cohen, Jeffrey Ira, Harriman, Kurt, Goldshuv, Alon, McCline, Matthew Clark, Eshleman, John Glenn, Lonergan, Luke
• Granted Patent: US 8,762,333 B2
• Time in Patent Office: Days
• US Class Current: 707/602
• CPC Class Codes: G06F 16/1865 Transactional file systems
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A computer readable storage medium includes executable instructions to store load data in new rows of an append only table within a data warehouse, without updating or deleting existing rows in the append only table. The new rows are added in a database transaction that guarantees that the distributed reads and writes of all transactions are processed reliably and that the transactions obey snapshot isolation rules. Data changes associated with the load data are maintained in a separate update table. Data from the append only table is merged with data changes from the update table to supply read data.

Claims

1. A computer readable storage medium, comprising executable instructions to:

store load data in new rows of an append only table within a data warehouse, without updating or deleting existing rows in the append only table;

maintain data changes associated with the load data in a separate update table; and

merge data from the append only table with data changes from the update table to supply read data.

2. The computer readable storage medium of claim 1 wherein the executable instructions to store include executable instructions to allow N different sources to store data to N different files concurrently.

3. The computer readable storage medium of claim 1 wherein the executable instructions to store include executable instructions to force a write of the load data to disk storage without a transaction log entry.

4. The computer readable storage medium of claim 1 further comprising executable instructions to strip transaction visibility information from the load data.

5. The computer readable storage medium of claim 4 wherein the executable instructions to strip transaction visibility information are executed by an Extract, Transform and Load module.

6. The computer readable storage medium of claim 4 wherein the executable instructions to strip transaction visibility information are executed at the data warehouse.

7. The computer readable storage medium of claim 1 wherein the load data is compressed.

8. The computer readable storage medium of claim 1 wherein the executable instructions are executed on a parallel partitioned table of a parallel data warehouse with multiple computational nodes.

9. The computer readable storage medium of claim 1 wherein the load data is received from an Extraction, Transform and Load module.

10. The computer readable storage medium of claim 1 wherein the data changes include data deletes and data updates.

11. The computer readable storage medium of claim 1 wherein the data changes are associated with multi-version concurrency control (MVCC) information.

12. The computer readable storage medium of claim 11 further comprising executable instructions to track data visibility in the append only table using the MVCC information.

13. The computer readable storage medium of claim 12 wherein the MVCC information specifies a highest committed row value indicative of an end of file.