Load-balancing and scaling for analytics data

• US 8,392,380 B2
• Filed: 07/30/2009
• Issued: 03/05/2013
• Est. Priority Date: 07/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhao, Mingyang, Kamani, Prashant
• Primary Examiner(s): Wilson, Kimberly
• Assistant Examiner(s): Uddin, Mohammed R
• Application Number:
  US12/512,776
• Publication Number:
  US 20110029581A1
• Time in Patent Office: 1,314 Days
• Field of Search: 707/688, 707/999.102
• US Class Current: 707/688
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: H04L 43/02 Capturing of monitoring data
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1014 based on the content of a request

Abstract

Load-balancing and scaling for analytics data may be provided. A logging system may receive data and select a stager database in which to store the data. The selection may be made according to an identifier associated with the data. The stored data may be processed and stored back to the stager database before being copied to a reporting database. The processed data may be aggregated with other data in the reporting database to provide an analytics report.

Claims

1. A method for providing data distribution, the method comprising:

receiving at least one element of data;

selecting one of a plurality of database servers, wherein each of the plurality of database servers is associated with a sequence number, according to an identifier associated with the at least one element of data wherein selecting the one of the plurality of database servers comprises computing a database sequence number by modulating an identifier associated with the at least one element of data by a total number of the plurality of database servers;

storing the at least one element of data in the selected database server;

processing a plurality of data elements stored in at least one of the plurality of database servers, wherein processing the plurality of data elements comprises;

dividing the plurality of data elements into at least one batch,

copying the at least one batch of data elements to an analytics server, and

analyzing the at least one batch of data elements according to an insight model;

determining whether an aggregation scope associated with the processed plurality of data elements is not assigned to at least one of a plurality of reporting databases; and

in response to determining that the aggregation scope is not assigned to at least one of a plurality of reporting database servers;

assigning the aggregation scope to at least one of the plurality of reporting database servers according to a potential data volume of the aggregation scope and a data capacity of each of the plurality of reporting database servers, and

aggregating the processed plurality of data elements into the at least one of the plurality of reporting database servers.

2. The method of claim 1, wherein the at least one element of data comprises data associated with a user interaction with a web site.

3. The method of claim 1, further comprising aggregating the processed data stored in the at least one reporting database server into a usage report.

4. The method of claim 3, further comprising exposing the usage report to at least one client application.

5. The method of claim 1, wherein the identifier associated with the at least one element of data comprises at least one of the following:

a user ID, a client-side cookie, and a browser session ID.

6. The method of claim 1, wherein processing the plurality of data elements stored in at least one of the plurality of database servers comprises at least one of the following:

removing an unwanted data element, calculating a behavior metric, and calculating a satisfaction classification.

7. The method of claim 1, wherein storing the processed data elements in at least one reporting database server comprises:

assigning each of a plurality of aggregation scopes to at least one of a plurality of reporting database servers comprising the at least one reporting database server;

copying the processed data elements to the at least one reporting database server according to the assigned aggregation scope; and

aggregating, by each of the plurality of reporting database servers, the processed data elements associated with the assigned aggregation scope for each of the plurality of reporting database servers into at least one usage report.

8. The method of claim 7, wherein the aggregation scope comprises at least one of the following:

an organization, a farm, a site, a sub-site, a page, and a web element.

9. The method of claim 7, wherein assigning each of a plurality of aggregation scopes to one of a plurality of reporting database servers comprises:

sorting the plurality of aggregation scopes in order of potential data volume;

sorting the plurality of reporting database servers in order of performance capability; and

iteratively assigning each of the sorted plurality of aggregation scopes to at least one of the sorted plurality of reporting database servers.

10. The method of claim 9, wherein sorting the plurality of reporting database servers in order of performance capability comprises sorting the plurality of reporting database servers according to a current data storage volume of each of the plurality of reporting database servers.

11. The method of claim 1, wherein dividing the plurality of data elements into the at least one batch, comprises associating the at least one batch with the identifier associated with at least one element of data of the plurality of data elements in the at least one batch.

12. A method for load-balancing analytics data comprising:

capturing a user behavior associated with a web site, wherein the user behavior is associated with a user behavior identifier;

selecting a raw data database from among a plurality of raw data databases, wherein selecting the raw data database comprises;

assigning a sequence number to each of the plurality of raw data databases,

calculating a database identifier from the user behavior identifier modulated by a total number of the plurality of raw databases, and

selecting the sequence number associated with the database identifier;

storing the captured user behavior in the selected raw data database;

processing the captured user behavior, wherein processing the captured behavior comprises;

dividing a plurality of captured behaviors into at least one batch,

copying the at least one batch of captured behaviors to an analytics server, and

analyzing the at least one batch of captured behaviors according to an insight model;

storing the processed user behavior in the selected raw data database;

determining whether any of a plurality of aggregation scopes are not assigned to at least one of a plurality of reporting databases; and

in response to determining that at least one of the plurality of aggregation scopes are not assigned to at least one of a plurality of reporting databases, assign the at least one unassigned scope to at least one of the plurality of reporting databases according to a potential data volume of the at least one unassigned scope and a data capacity of each of the plurality of reporting databases.

13. The method of claim 12, further comprising assigning the processed user behavior to a reporting database.

14. The method of claim 13, further comprising aggregating the processed user behavior with at least one second processed user behavior.

15. The method of claim 13, wherein the processed user behavior is assigned to the reporting database according to at least one of the plurality of aggregation scopes.

16. The method of claim 15, wherein the at least one of the plurality of aggregation scopes comprise at least one of the following:

a site farm, an application, a site-collection, a web site, and a web page.

17. The method of claim 12, wherein the raw data database is selected according to an identifier associated with the user behavior.

18. The method of claim 12, wherein processing the captured user behavior comprises batching a plurality of captured behaviors from a user after a configurable time of inactivity from the user.

19. A system for providing load-balancing and scaling for data distribution, the system comprising:

a memory storage; and

a processing unit coupled to the memory storage, wherein the processing unit is configured to;

capture a user behavior associated with a web site, wherein the user behavior is associated with an identifier comprising at least one of the following;

a user ID and a browser session ID;

select a raw database from among a plurality of raw databases, wherein being configured to select the raw database comprises being configured to;

assign a sequence number to each of the plurality of raw databases,

calculate a database identifier from the user behavior identifier modulated by a total number of the plurality of raw databases, and

select the sequence number associated with the database identifier;

store the captured user behavior in the selected raw database;

process the captured user behavior, wherein being configured to process the user behavior comprises being configured to;

divide a plurality of captured user behaviors into a batch,

copy the batch of captured user behaviors to an analytics server, and

analyze the batch of captured user behaviors according to a behavior insight model;

store the processed user behavior in the selected raw data database;

determine whether any of a plurality of aggregation scopes are not assigned to at least one of a plurality of reporting databases;

in response to determining that at least one of the plurality of aggregation scopes are not assigned to at least one of a plurality of reporting databases, assign the at least one unassigned scope to at least one of the plurality of reporting databases according to a potential data volume of the at least one unassigned scope and a data capacity of each of the plurality of reporting databases;

copy the processed user behavior to one of the plurality of reporting databases according to the scope assigned to the one of the reporting databases; and

aggregate the copied user behavior with at least one other copied user behavior of the same type into a usage report wherein the user behavior type comprises at least one of the following;

a click-through, a search, a unique visitor count, and a dwell time.