Data transfer optimization

• US 8,874,620 B1
• Filed: 06/15/2011
• Issued: 10/28/2014
• Est. Priority Date: 06/15/2011
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Rohr, Vincent M., Cook, Brandon A.
• Primary Examiner(s): Mofiz, Apu
• Assistant Examiner(s): Khoshnoodi, Fariborz
• Application Number:
  US13/161,385
• Time in Patent Office: 1,231 Days
• Field of Search: 707/803, 707/809
• US Class Current: 707/803
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application

Abstract

A dataset structure that is suitable for transferring data between service components may include a table having one or more rows and a plurality of columns. The table may be defined by a schema for an entity type. The schema may include a corresponding column definition for each column. Each of the rows may store a tuple of information that conforms to the entity type defined by the schema. A data transfer component may use the dataset structure to transfer data between the service components.

Claims

1. A computer-implemented method, comprising:

under control of one or more servers configured with executable instructions,

storing data in a table having a plurality of rows and a plurality of columns, individual rows storing a tuple of information that conforms to an entity type defined by a schema, the schema including a corresponding column definition for individual columns;

storing the plurality of rows in the table that have a variable row size or a fixed row size into a plurality of pages, at least one row of data in a page having the variable row size being positioned using a corresponding unique absolute offset value and at least one corresponding unique column offset value, and at least one row of data in a page having the fixed row size being positioned using a corresponding unique absolute offset value and at least one common column offset value that is identical for the plurality of rows; and

modifying a particular row to include at least one more byte of data or at least one less byte of data;

adjusting an absolute offset of an additional row that follows the particular row based on a change in an amount of bytes in the particular row due to modification of the particular row while maintaining a relative offset of the additional row as unchanged; and

updating a page that includes the particular row with a page revision number that indicates that the particular row of the page is modified.

2. The computer-implemented method of claim 1, wherein the page has a unique identifier and a row in the plurality of pages has a unique identifier that is permanently associated with the row, further comprising checking the page for a specific row in response to a requested to access the specific row even after the specific row has been deleted from the page.

3. The computer-implemented method of claim 1, further comprising transferring data from a first service component to a second service component by differentially serializing and transferring one or more of the plurality of pages, the differentially serializing including serializing and transferring a first page of the plurality of pages in response to determining that the first page includes one or more modified rows of data and passing over a second page of the plurality of pages from serialization and transfer in response to determining that the second page does not include any modified rows of data.

4. A system, comprising:

one or more processors; and

a memory that includes a plurality of components, the components comprising;

a dataset structure that includes a table having multiple rows and a plurality of columns, the table being defined by a schema for an entity type that includes a corresponding column definition for individual columns, and the at least one row storing a tuple of information that conforms to the entity type defined by the schema; and

a data transfer component that transfers data stored in the dataset structure between service components and between component modules in a pipeline of a service component, the data transfer component transferring the data stored in the table through the pipeline by propagating data stored in a first row of the table to a first component module of the pipeline for processing while propagating data stored in a second row of the table to a second component module of the pipeline for processing.

5. The system of claim 4, further comprising a serializer that converts at least some of the data in the dataset structure to another format for transmission over a network, the serializer performing a full serialization that converts every portion of the data stored in the dataset structure or a differential serialization that converts a modified portion of the data stored in the dataset structure.

6. The system of claim 5, wherein the full serialization or the differential serialization generates formatted data that includes one or more bytes that designate the formatted data as organized according to the dataset structure, and wherein a first additional byte indicates a major version of a serializer and a second additional byte indicates a minor version of the serializer.

7. The system of claim 5, wherein the differential serialization converts data included in a first page of data for transmission over a network in response to determining that the first page includes at least one piece of modified information and passes over a second page of data from undergoing data conversion in response to determining that the second page does not include any piece of modified information.

8. The system of claim 4, wherein the service components process the data stored in the dataset structure for an application.

9. The system of claim 4, wherein the table includes a plurality of rows that are organized into pages, the pages having unique identifiers and individual rows in a page having a unique identifier for a corresponding page.

10. The system of claim 9, wherein the page of the pages includes a recommended size that is exceeded during an addition of data to the page.

11. The system of claim 9, wherein a row is stored on the page of the pages such that the data transfer component, in response to a request to interact with the row, checks the page for the row even after deletion of the row.

12. The system of claim 9, wherein the pages includes a fixed row size page, and wherein the data transfer component locates data from a row of multiple rows of data in the fixed row size page using a corresponding unique absolute offset value and at least one common column offset value that is identical for all of the multiple rows.

13. The system of claim 9, wherein the pages includes a variable row size page, and wherein the data transfer component locates data from a row of multiple rows of data in the variable row size page using a corresponding unique absolute offset value and at least one corresponding unique column offset value.

14. The system of claim 4, wherein a column definition for a corresponding column designate the column as including an attribute, an attribute name of the attribute, an attribute type of the attribute, or whether the attribute is a primary key.

15. The system of claim 4, wherein a query for data from one or more rows of the table is executed using custom code without use of pre-established query language.

16. The system of claim 4, wherein the dataset structure includes an additional table of the entity type that shares an identical schema as the table, the additional table having additional column definitions that are identical to the column definitions of the table.

17. The system of claim 4, wherein a column includes a value type that exists with a specific scope or as reference type that is referenced with a pointer.

18. One or more non-transitory computer readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

inserting a replacement row for an existing row of an original page into a replacement page, the replacement row having a different byte length than the existing row;

copying a subsequent row that follows the existing row from an original page to a replacement page, the subsequent row having an absolute offset that indicates a beginning position of the subsequent row in the original page and a relative offset that indicates a beginning position of a column in the subsequent row;

providing the replacement page with an identifier of the original page;

assigning the replacement page with a value flag that indicates the replacement page includes modified data; and

adjusting the absolute offset of the subsequent row based on a byte length difference between the replacement row and the existing row while maintaining the relative offset as unchanged.

19. The one or more non-transitory computer readable media of claim 18, further comprising instructions that, when executed, cause the one or more processors to perform an act of deleting the original page.

20. The one or more non-transitory computer readable media of claim 18, further comprising instructions that, when executed, cause the one or more processors to perform an act of providing the replacement page with at least one of a revision number that indicates the replacement page is more recent than the original page.

21. The one or more non-transitory computer readable media of claim 18, wherein the absolute offset and the relative offset are stored in at least one index of the replacement page.

22. The one or more non-transitory computer readable media of claim 18, wherein the original page and the replacement page store data in a table of a dataset structure, the table being defined by a schema for an entity type, and at least one of the replacement row and the subsequent row storing a corresponding tuple of information that conforms to an entity type defined by the schema.

23. The one or more non-transitory computer readable media of claim 22, wherein the schema includes a corresponding column definition for a column in the replacement row or the subsequent row.

24. A computer-implemented method, comprising:

under control of one or more computing systems configured with executable instructions,

determining whether a data processing scenario includes use of a variable size data tuple;

storing one or more data tuples used in the data processing scenario in corresponding rows of a variable row size table in response to a determination that the data processing scenario includes the use of the at least one variable size data tuple; and

storing the one or more data tuples used in the data processing scenario in corresponding rows of a fixed row size table in response to a determination that the data processing scenario lacks the use of the at least one variable size data tuple.

25. The computer-implemented method of claim 24, wherein performing a transaction on a data tuple in the variable row size table uses more processing power than performing a transaction on a data tuple in the fixed row size table.

26. The computer-implemented method of claim 24, wherein a row of data tuple of a plurality of rows of data tuples in the variable row size table is positioned using a corresponding unique absolute offset value and at least one corresponding unique column offset value, and wherein a row of data tuple in the fixed row size table is positioned using a corresponding unique absolute offset value and at least one common column offset value that is identical for a plurality of rows of data tuples.

27. The computer-implemented method of claim 24, wherein the fixed row size table takes more bandwidth to transfer between a plurality of service components than the variable row size table.

28. The computer-implemented method of claim 24, wherein a first index that stores a corresponding absolute offset value and at least one corresponding unique column offset value for individual rows in the variable row size table is bigger in size than a second index that store the corresponding unique absolute offset value and at least one common column offset value for individual rows in the fixed row size table.