Load-Balancing and Scaling for Analytics Data

• US 20110029581A1
• Filed: 07/30/2009
• Published: 02/03/2011
• Est. Priority Date: 07/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhao, Mingyang, Kamani, Prashant
• Granted Patent: US 8,392,380 B2
• Time in Patent Office: Days
• US Class Current: 707/812
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: H04L 43/02 Capturing of monitoring data
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1014 based on the content of a request

Abstract

Load-balancing and scaling for analytics data may be provided. A logging system may receive data and select a stager database in which to store the data. The selection may be made according to an identifier associated with the data. The stored data may be processed and stored back to the stager database before being copied to a reporting database. The processed data may be aggregated with other data in the reporting database to provide an analytics report.

Claims

1. A method for providing data distribution, the method comprising:

receiving at least one element of data;

selecting one of a plurality of database servers according to an identifier associated with the at least one element of data;

storing the at least one element of data in the selected database server;

processing a plurality of data elements stored in at least one of the plurality of database servers; and

aggregating the processed data elements in at least one reporting database server.

2. The method of claim 1, wherein the at least one element of data comprises data associated with a user interaction with a web site.

3. The method of claim 1, further comprising aggregating the processed data stored in the at least one reporting database server into a usage report.

4. The method of claim 3, further comprising exposing the usage report to at least one client application.

5. The method of claim 1, wherein the identifier associated with the at least one element of data comprises at least one of the following:

a user ID, a client-side cookie, and a browser session ID.

6. The method of claim 1, wherein selecting one of a plurality of database servers according to an identifier associated with the at least one element of data comprises selecting a number associated with the one of the plurality of database values by modulating an integer hash of the identifier associated with the at least one element of data by a total number of database servers of the plurality of database servers.

7. The method of claim 1, wherein processing the plurality of data elements stored in at least one of the plurality of database servers comprises at least one of the following:

removing an unwanted data element, calculating a behavior metric, and calculating a satisfaction classification.

8. The method of claim 1, wherein storing the processed data elements in at least one reporting database server comprises:

assigning each of a plurality of aggregation scopes to at least one of a plurality of reporting database servers comprising the at least one reporting database server;

copying the processed data elements to the at least one reporting database server according to the assigned aggregation scope; and

aggregating, by each of the plurality of reporting database servers, the processed data elements associated with the assigned aggregation scope for each of the plurality of reporting database servers into at least one usage report.

9. The method of claim 8, wherein the aggregation scope comprises at least one of the following:

an organization, a farm, a site, a sub-site, a page, and a web element.

10. The method of claim 8, wherein assigning each of a plurality of aggregation scopes to one of a plurality of reporting database servers comprises:

sorting the plurality of aggregation scopes in order of potential data volume;

sorting the plurality of reporting database servers in order of performance capability; and

iteratively assigning each of the sorted plurality of aggregation scopes to at least one of the sorted plurality of reporting database servers.

11. The method of claim 10, wherein sorting the plurality of reporting database servers in order of performance capability comprises sorting the plurality of reporting database servers according to a current data storage volume of each of the plurality of reporting database servers.

12. The method of claim 1, further comprising dividing data stored in one of a plurality of database servers into a plurality of processing batches, wherein each of the processing batches is associated with the identifier associated with the at least one element of data.

13. A computer-readable medium which stores a set of instructions which when executed performs a method for load-balancing analytics data, the method executed by the set of instructions comprising:

capturing a user behavior associated with a web site;

selecting a raw data database from among a plurality of raw data databases;

storing the captured user behavior in the selected raw data database;

processing the captured user behavior; and

storing the processed user behavior in the selected raw data database.

14. The computer-readable medium of claim 13, further comprising assigning the processed user behavior to a reporting database.

15. The computer-readable medium of claim 14, further comprising aggregating the processed user behavior with at least one second processed user behavior.

16. The computer-readable medium of claim 14, wherein the processed user behavior is assigned to the reporting database according to an aggregation scope.

17. The computer-readable medium of claim 16, wherein the aggregation scope comprises at least one of the following:

a site farm, an application, a site-collection, a web site, and a web page.

18. The computer-readable medium of claim 13, wherein the raw data database is selected according to an identifier associated with the user behavior.

19. The computer-readable medium of claim 13, wherein processing the captured user behavior comprises batching a plurality of captured behaviors from a user after a configurable time of inactivity from the user.

20. A system for providing load-balancing and scaling for data distribution, the system comprising:

a memory storage; and

a processing unit coupled to the memory storage, wherein the processing unit is operative to;

capture a user behavior associated with a web site, wherein the user behavior is associated with an identifier comprising at least one of the following;

a user ID and a browser session ID;

select a raw database from among a plurality of raw databases, wherein being operative to select the raw database comprises being operative to;

assign a sequence number to each of the plurality of raw databases,

calculate a database identifier from the user behavior identifier modulated by a total number of the plurality of raw databases, and

select the sequence number associated with the database identifier;

store the captured user behavior in the selected raw database;

process the captured user behavior, wherein being operative to process the user behavior comprises being operative to;

divide a plurality of captured user behaviors into a batch,

copy the batch of captured user behaviors to an analytics server, and

analyze the batch of captured user behaviors according to a behavior insight model;

store the processed user behavior in the selected raw data database;

determine whether any of a plurality of aggregation scopes are not assigned to at least one of a plurality of reporting databases;

in response to determining that at least one of the plurality of aggregation scopes are not assigned to at least one of a plurality of reporting databases, assign the at least one unassigned scope to at least one of the plurality of reporting databases according to a potential data volume of the at least one unassigned scope and a data capacity of each of the plurality of reporting databases;

copy the processed user behavior to one of the plurality of reporting databases according to the scope assigned to the one of the reporting databases; and

aggregate the copied user behavior with at least one other copied user behavior of the same type into a usage report wherein the user behavior type comprises at least one of the following;

a click-through, a search, a unique visitor count, and a dwell time.