Systems and Methods for Data Manipulation Using Multiple Storage Formats

• US 20070276786A1
• Filed: 05/15/2006
• Published: 11/29/2007
• Est. Priority Date: 05/15/2006
• Status: Active Grant

• Current Assignee: Algebraix Data Corp.
• Original Assignee: Algebraix Data Corp.
• Inventors: Piedmonte, Christopher
• Granted Patent: US 7,720,806 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2453 Query optimisation

Abstract

Systems and methods for storing and accessing data. Algebraic relations may be composed that each define a result equal to a requested data set. The algebraic relations may reference other data sets in storage. Some of the data sets may contain the same logical data stored in different physical formats and/or in different locations in the data store. One of the algebraic relations may be selected for use in providing the requested data set based, at least in part, on the physical format and/or locations of the data sets referenced in the algebraic relations. In other examples, algebraic relations may be selected based, at least in part, on the speed and available bandwidth of the channel(s) used to retrieve data sets referenced in the algebraic relation. Functions may be used to calculate the algebraic relation using the data sets retrieved from storage. The functions may be format specific based on the physical formats of the data sets provided as operands to the functions.

Claims

1. A method for providing a requested data set:

providing a storage system for storing data sets;

composing at least two alternative algebraic relations that each define a result equal to the requested data set, including a first algebraic relation based on at least a first data set stored in the storage system and a second algebraic relation based on at least a second data set stored in the storage system;

wherein the first data set and the second data set contain the same logical data stored in different physical formats;

selecting between the first algebraic relation and the second algebraic relation based, at least in part, on the physical format of the first data set and the physical format of the second data set; and

using the selected algebraic relation to calculate the requested data set.

2. The method of claim 1, wherein the physical format of the first data set is selected from the group consisting of comma separated value format, tab delimited value format, binary string format, fixed offset format and markup language format.

3. The method of claim 2, wherein the physical format of the second data set is selected from the group consisting of comma separated value format, tab delimited value format, binary string format, fixed offset format and markup language format.

4. The method of claim 1, further comprising providing plurality of functions for calculating algebraic relations, wherein the plurality of functions includes a first function that operates on a data set in the physical format of the first data set and a second function that operates on a data set in the physical format of the second data set, wherein the first function and the second function are algebraically equivalent;

wherein the step of using the selected algebraic relation to calculate the requested data set further comprises using the first function if the first algebraic relation is selected and using the second function if the second algebraic relation is selected.

5. The method of claim 1, wherein the storage system includes a plurality of data channels and the first data set and the second data set are available over different data channels.

6. The method of claim 5, wherein the first data set in the first physical format has a first size and is available over a first data channel having a first data transfer rate, and the second data set in the second physical format has a second size and is available over a second data channel having a second data transfer rate, the method further comprising:

determining a first cost based, at least in part, on the first size and the first data transfer rate;

determining a second cost based, at least in part, on the second size and the second data transfer rate; and

selecting the first algebraic relation to calculate the requested data set if the first cost is lower than the second cost.

7. A method for providing a requested data set:

providing a storage system for storing data sets, wherein the storage system includes a volatile memory and a non-volatile storage medium;

composing at least two alternative algebraic relations that each define a result equal to the requested data set, including a first algebraic relation based on at least a first data set stored in the storage system and a second algebraic relation based on at least a second data set stored in the storage system;

wherein the first data set and the second data set contain the same logical data stored in different physical formats, the first data set being stored in a first physical format having a first size and the second data being stored in a second physical format having a second size that is smaller than the first size;

determining that the first data set is available in the volatile memory; and

selecting the first algebraic relation to calculate the requested data set.

8. A method for providing a requested data set:

providing a storage system for storing data sets in a plurality of physical formats;

providing a plurality of functions for operating on the data sets in the plurality of physical formats, wherein each function is configured to operate on at least one operand in a respective one of the physical formats;

providing an algebraic relation that defines a result equal to the requested data set, wherein the algebraic relation references at least a first data set stored in the storage system;

retrieving the first data set in a respective one of the physical formats;

selecting one of the plurality of functions that is configured to operate on an operand in the respective physical format of the first data set; and

using the selected function to calculate the requested data set from the algebraic relation.

9. The method of claim 8, wherein the algebraic relation specifies at least one operation and the selected function corresponds to the specified operation.

10. The method of claim 8, wherein the plurality of functions includes algebraically equivalent functions configured to operate on operands in different physical formats.

11. The method of claim 8, wherein the step of providing the algebraic relation further comprises selecting the algebraic relation from a plurality of algebraic relations that each define a result equal to the requested data set.

12. The method of claim 10, wherein the step of providing the algebraic relation further comprises selecting the algebraic relation from a plurality of algebraic relations that each define a result equal to the requested data set.

13. The method of claim 12, wherein the step of selecting the algebraic relation further comprises selecting the algebraic relation based, at least in part, on an estimate of the time required to retrieve the first data set from the storage system.

14. The method of claim 8, wherein the algebraic relation references at least a second data set stored in the storage system, the method further comprising retrieving the second data set in a respective one of the physical formats.

15. The method of claim 14, wherein the selected function is configured to operate on at least a second operand in the respective physical format of the second data set.

16. The method of claim 14, wherein the algebraic relation specifies at least one operation and the selected function corresponds to the specified operation.

17. The method of claim 14, wherein the plurality of functions includes algebraically equivalent functions configured to operate on operands in different physical formats.

18. The method of claim 14, wherein the step of providing the algebraic relation further comprises selecting the algebraic relation from a plurality of algebraic relations that each define a result equal to the requested data set.

19. The method of claim 18, wherein the step of selecting the algebraic relation further comprises selecting the algebraic relation based, at least in part, on an estimate of the time required to retrieve the second data set from the storage system.

20. The method of claim 17, wherein the step of providing the algebraic relation further comprises selecting the algebraic relation from a plurality of algebraic relations that each define a result equal to the requested data set.