Systems and methods for transforming data in buffer memory without unnecessarily copying data to additional memory locations

• US 20040186843A1
• Filed: 10/08/2003
• Published: 09/23/2004
• Est. Priority Date: 03/18/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Blaszczak, Michael A.
• Granted Patent: US 7,860,916 B2
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: G06F 12/023 Free address space management

Abstract

Various embodiments of the present invention are directed to a unique memory management scheme utilized by the DTP whereby data extracted from an external source is placed in a memory buffer and is then manipulated by the components without the need for copying the data to any other location in memory. While logically the data moves from component to component in the DFE for the various embodiments described herein, the data does not in fact change locations but, instead, the data resides in the buffer and is operated upon by a series of components that, in turn, access the data in the buffer via pointers and manipulate same.

Claims

1. A method for transforming data comprising:

extracting data from at least one external data source;

storing the data in a buffer;

establishing a first set of pointers to the data;

passing the first set of pointers to the data in the buffer to a first component in order for the first component to transform the data directly in the buffer; and

loading the data from the buffer to at least one external data destination.

2. The method of claim 1 wherein the data extracted from an external data source is a table of data comprising at least one row, each said row comprising at least one column.

3. The method of claim 2 wherein a memory location corresponding to a start of a specific row is determined as a function of a row reference number and a row width indicative of a memory offset corresponding to said start of said specific row.

4. The method of claim 2 wherein a memory location corresponding to a start of a specific column in a specific row is determined as a function of a row reference number and a row width plus a column offset indicative of a memory offset corresponding to said start of said specific column in said specific row.

5. The method of claim 2 wherein the first set of pointers point to the beginning of the rows.

6. The method of claim 5 wherein the step of establishing first set of pointers that point to the beginning of the rows comprising the sub-step of determining the beginning of a row as a function of the row number and the row width.

7. The method of claim 5 further comprising, after the element of passing the first set of pointers to the data in the buffer to a first component in order for the first component to transform the data directly in the buffer, passing the first set of pointers to the data in the buffer to a subsequent component in order for the subsequent component to transform the data directly in the buffer.

8. The method of claim 5 further comprising, after the element of storing the data in a buffer and establishing a first set of pointers to the data, establishing a second set of pointers to the data and establishing a third set of pointers to the data.

9. The method of claim 8 wherein said method further comprises, after the element of passing the first set of pointers to the first component, passing the second set of pointers and the third set of pointers to the first component.

10. The method of claim 9 further comprising a split method by which the first component splits the data, said split method comprising:

the first component receiving the first set of pointers, the second set of pointers, and the third set of pointers;

the first component traversing each row via the first set of pointers;

for each row, the first component designating each row as either a first path row or a second path row based on a criteria for splitting said data;

for each first path row, assigning a pointer from the second set of pointers to point at each such first path row;

for each second path row, assigning a pointer from the third set of pointers to point at each such second path row; and

returning the second set of pointers and the third set of pointers.

11. A method for transforming data comprising:

extracting data from a source, said data comprising a plurality of rows;

writing the data to a buffer;

creating a plurality of pointers wherein each pointer uniquely points to a single row of data from among the plurality of rows of data in the buffer;

passing the plurality of pointers to a plurality of subsequent transformation object in a path, the first transformation object being the first transformation object on the first pass, the second transformation object on the second pass, and so forth;

enabling the transformation object to transform the data in the buffer, said transformation object directly accessing the data in the buffer via the pointers;

returning to the element of passing the plurality of pointers to a subsequent transformation if there remains any transformations unexecuted in the path;

reading the data from the buffer; and

loading the data to a destination.

12. The method of claim 11 wherein the act of enabling the transformation object to transform the data in the buffer comprises the modification of a value in a data cell.

13. The method of claim 11 wherein the act of enabling the transformation object to transform the data in the buffer comprises the swapping of at least two pointers.

14. The method of claim 13 wherein the transformation object transforms the data by sorting the data via swapping at least two pointers.

15. The method of claim 13 wherein the transformation object transforms the data by initializing at least two more arrays to point to select elements of said data.

16. A computer-readable medium bearing computer-readable instructions for:

extracting data from at least one external data source;

storing the data in a buffer;

establishing a first set of pointers to the data;

passing the first set of pointers to the data in the buffer to a first component in order for the first component to transform the data directly in the buffer; and

loading the data from the buffer to at least one external data destination.

17. The computer-readable medium of claim 16 further comprising computer-readable instructions wherein the data extracted from an external data source is a table of data comprising at least one row, each said row comprising at least one column.

18. The computer-readable medium of claim 17 further comprising computer-readable instructions wherein the first set of pointers point to the beginning of the rows.

19. The computer-readable medium of claim 18 further comprising computer-readable instructions for, after the element of passing the first set of pointers to the data in the buffer to a first component in order for the first component to transform the data directly in the buffer, passing the first set of pointers to the data in the buffer to a subsequent component in order for the subsequent component to transform the data directly in the buffer.

20. The computer-readable medium of claim 18 further comprising computer-readable instructions for, after the element of storing the data in a buffer and establishing a first set of pointers to the data, establishing a second set of pointers to the data and establishing a third set of pointers to the data.

21. The computer-readable medium of claim 20 further comprising, after the element of passing the first set of pointers to the first component, passing the second set of pointers and the third set of pointers to the first component.

22. The computer-readable medium of claim 21 further comprising computer-readable instructions for a split method by which the first component splits the data, said split method comprising:

the first component receiving the first set of pointers, the second set of pointers, and the third set of pointers;

the first component traversing each row via the first set of pointers;

for each row, the first component designating each row as either a first path row or a second path row based on a criteria for splitting said data;

for each first path row, assigning a pointer from the second set of pointers to point at each such first path row;

for each second path row, assigning a pointer from the third set of pointers to point at each such second path row; and

returning the second set of pointers and the third set of pointers.

23. A system comprising a processor, memory, and instructions for:

extracting data from at least one external data source;

storing the data in a buffer;

establishing a first set of pointers to the data;

passing the first set of pointers to the data in the buffer to a first component in order for the first component to transform the data directly in the buffer; and

loading the data from the buffer to at least one external data destination.

24. The system of claim 23 further comprising computer-readable instructions wherein the data extracted from an external data source is a table of data comprising at least one row, each said row comprising at least one column.

25. The system of claim 24 further comprising computer-readable instructions wherein the first set of pointers point to the beginning of the rows.

26. The system of claim 25 further comprising computer-readable instructions for, after the element of passing the first set of pointers to the data in the buffer to a first component in order for the first component to transform the data directly in the buffer, passing the first set of pointers to the data in the buffer to a subsequent component in order for the subsequent component to transform the data directly in the buffer.

27. The system of claim 25 further comprising computer-readable instructions for, after the element of storing the data in a buffer and establishing a first set of pointers to the data, establishing a second set of pointers to the data and establishing a third set of pointers to the data.

28. The system of claim 27 further comprising, after the element of passing the first set of pointers to the first component, passing the second set of pointers and the third set of pointers to the first component.

29. The system of claim 28 further comprising computer-readable instructions for a split method by which the first component splits the data, said split method comprising:

the first component receiving the first set of pointers, the second set of pointers, and the third set of pointers;

the first component traversing each row via the first set of pointers;

for each row, the first component designating each row as either a first path row or a second path row based on a criteria for splitting said data;

for each first path row, assigning a pointer from the second set of pointers to point at each such first path row;

for each second path row, assigning a pointer from the third set of pointers to point at each such second path row; and

returning the second set of pointers and the third set of pointers.