Batched Transfer of Arbitrarily Distributed Data

• US 20100332531A1
• Filed: 06/26/2009
• Published: 12/30/2010
• Est. Priority Date: 06/26/2009
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Galande, Ashutosh
• Application Number:
  US12/492,675
• Publication Number:
  US 20100332531A1
• Time in Patent Office: Days
• US Class Current: 707/770
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

Batched data transfer may be provided. Data, such as logging data amassed by a web site server, may be distributed among a plurality of database servers. A client may request access to some and/or all of that data. The amount of data may require dividing the data into batches comprising a portion of data from each of the database servers. The batches of data may be transferred to the client and may each cover a specified period of time and a state variable may be used to store the maximum time of data that has been received by the client.

Claims

1. A method for providing batched data transfers, the method comprising:

distributing data among a plurality of database servers;

receiving a request for at least a portion of the data;

determining whether the requested portion of the data requires batched delivery;

in response to determining that the requested portion of the data requires batched delivery, dividing the requested portion of the data into a plurality of batches; and

providing a first batch of the requested portion of the data.

2. The method of claim 1, wherein distributing the data among the plurality of database servers comprises allocating data to the plurality of database servers in a round-robin distribution.

3. The method of claim 1, wherein distributing the data among the plurality of database servers comprises allocating data to at least one of the plurality of database servers according to an available capacity of the at least one database server.

4. The method of claim 1, wherein the database servers share a common database schema.

5. The method of claim 1, wherein the data is ordered by a storage time.

6. The method of claim 1, further comprising establishing a context between a requesting user and the plurality of database servers.

7. The method of claim 6, wherein establishing the context comprises specifying at least one type of data requested.

8. The method of claim 6, wherein establishing the context comprises identifying an amount of data available.

9. The method of claim 6, wherein establishing the context comprises identifying at least one of the following:

a minimum time and a maximum time for which data is available.

10. The method of claim 1, further comprising establishing a state variable associated with the batched data transfer.

11. The method of claim 10, wherein the state variable is assigned to a client associated with the data request.

12. The method of claim 11, wherein the state variable identifies a maximum time for which data has been transferred to the client.

13. The method of claim 11, further comprising:

saving the state variable after completion of the batched data transfer; and

reusing the state variable for at least one subsequent data request.

14. The method of claim 13, wherein the state variable is saved to at least one of the following locations:

the client and at least one of the plurality of database servers.

15. A computer-readable medium which stores a set of instructions which when executed performs a method for providing batched data transfers, the method executed by the set of instructions comprising:

receiving a data transfer request from a client device for data stored in a distributed database system;

establishing a transfer context with the client device;

determining whether a state variable associated with a previous data transfer to the client device is saved;

in response to determining that a state variable associated with a previous data transfer to the client device is not saved, creating a new state variable associated with the data request;

dividing the requested data into a plurality of batches; and

transferring the first batch of the requested portion of the data to the client device.

16. The computer-readable medium of claim 15, wherein establishing the transfer context with the client device comprises establishing a batch size to be used in dividing the requested data into the plurality of batches.

17. The computer-readable medium of claim 16, wherein the batch size is established according to at least one of the following:

an available data transfer bandwidth, a number of database rows, a default storage size, and a storage size requested by the client device.

18. The computer-readable medium of claim 15, wherein the state variable comprises a maximum data time of data received by the client device.

19. The computer-readable medium of claim 18, further comprising:

in response to determining that a state variable associated with a previous data transfer to the client device is saved, resuming the batched data transfer according to the maximum data time of the data received by the client device.

20. A system for providing batched data transfer, the system comprising:

a memory storage; and

a processing unit coupled to the memory storage, wherein the processing unit is operative to;

receive a data transfer request from a client device for data stored in a distributed database system, wherein the distributed database system comprises a plurality of stager databases sharing a common database schema and operative to;

store a plurality of analytics data according to a distribution algorithm, and

order the plurality of analytics data across the plurality of stager databases according to an event time associated with each of the plurality of analytics data,

establish a transfer context with the client device, wherein the transfer context comprises at least one of the following;

a type of requested data and a data batch size,

determine whether a state variable comprising a maximum time associated with a previous data transfer to the client device is saved,

in response to determining that a state variable associated with a previous data transfer to the client device is not saved, create a new state variable associated with the data request,

divide the requested data into a plurality of batches according to the data batch size,

transfer a first batch of the requested portion of the data to the client device,

wait for a request from the client device to send a second batch of the requested portion of the data, and

in response to receiving the request from the client device, transfer the second batch of the requested portion of the data to the client device.