DETERMINATION OF SAMPLING CHARACTERISTICS BASED ON AVAILABLE MEMORY

• US 20080282020A1
• Filed: 05/09/2007
• Published: 11/13/2008
• Est. Priority Date: 05/09/2007
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Shanbhag, Amit Umesh, Burgess, David A., Ames, Glen Anthony, Rajagopalan, Sundara Raman, Selle, Brian T.
• Granted Patent: US 7,818,534 B2
• Time in Patent Office: Days
• US Class Current: 711/100
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A portion of data records of a full input data set are imported into memory of a computer system for processing by an executing application. The full input data set includes data records of a dimensionally-modeled fact collection. An amount of the data of the full input set to import is determined based on an amount of available memory of the computer system. The sampling characteristics for sampling the full input data set are determined based on the amount of the data that can be imported and on characteristics of the full input data set and application involved. The full input data set is then sampled and a portion of the records are imported into the memory of the computer system for processing. The sampling characteristics are determined such that analysis as a result of processing by the executing application of the sampled portion of the records imported is representative of the analysis that could otherwise be carried out on the full input data set, with a calculable statistical relevance.

Claims

1. A method of importing a portion of data records of a full input data set into memory of a computer system for processing by an executing application, wherein the full input data set includes data records of a dimensionally-modeled fact collection, the method comprising:

determining an amount of the data of the full input set to import based on an amount of available memory of the computer system;

based on the determined amount of the data to import and on characteristics of the full input data set at least other than the total size of the full input data set, determining sampling characteristics for sampling the full input data set; and

causing a portion of the records of the full input data set to be imported into the memory of the computer system, including sampling the full input data set, to determine the portion of the records to import, in accordance with the determined sampling characteristics, wherein the sampling characteristics are determined such that analysis as a result of processing by the executing application of the sampled portion of the records imported is representative of the analysis that could otherwise be carried out on the full input data set, with a calculable statistical relevance.

2. The method of claim 1, wherein:

sampling the full input data set, to determine the portion of the records to import, includes sampling such that, for a particular dimension of the fact collection, for each record of the full input data set sampled, wherein that input record has a particular value at the particular dimension, every record of the full input data set having that particular value at the particular dimension is sampled.

3. The method of claim 1, wherein:

sampling the full input data set, to determine the portion of the records to import, includes applying a Fowler/Noll/Vo (FNV) Hash algorithm for sampling based on a key that is the value for each record at a particular dimension.

4. The method of claim 1, wherein:

each data record includes information regarding behavior of a particular user with respect to online services.

5. The method of claim 1, wherein:

sampling the full input data set, to determine the portion of the records to import, includes sampling such that, for each record of the full input data set sampled, wherein that input record has information about a particular entity, every record of the full input data set including information regarding behavior of the particular entity is sampled.

6. The method of claim 5, wherein:

the particular entity is one of a group consisting of user, account and application.

7. The method of claim 1, further comprising:

analyzing the sampled data records, including determining a statistical confidence for the analysis; and

causing a result of the analysis, including the determined statistical confidence, to be made available to a user.

8. The method of claim 1, wherein:

determining an amount of the data of the full input set to import based on an amount of available memory of the computer system includes determining the amount of the data in memory to be less than the amount of available memory of the computer system.

9. The method of claim 1, wherein:

sampling the full input data set, to determine the portion of the records to import, includes

assigning the full input data set into a plurality of bins, wherein for each record of the full input data set sampled, that input record has a particular value at a particular dimension and every record of the full input data set having the particular value at the particular dimension is in that same bin; and

sampling all the records of at least one of the bins.

10. A method of importing a portion of data records of a full input data set into memory of a computer system for processing by an executing application, wherein the full input data set includes data records of a dimensionally-modeled fact collection, the method comprising:

determining a nominal amount of the data of the full input set to import based on an amount of available memory of the computer system;

based on the determined amount of the data to import and on characteristics of the full input data set at least other than the total size of the full input data set, determining sampling characteristics for sampling the full input data set;

determining a characteristic associated with the determined amount of data;

adjusting the determined sampling characteristics based on a user-provided indication of a desired characteristic different from the determined characteristic associated with the determined amount of data;

causing a portion of the records of the full input data set to be imported into the memory of the computer system, including sampling the full input data set, to determine the portion of the records to import, in accordance with the adjusted sampling characteristics.

11. The method of claim 10, wherein:

the adjusted sampling characteristics include characteristics corresponding to inputting the portion of records a sub-portion at a time, wherein analysis is carried out on the imported sub-portions while the others of the sub-portions are not present in the memory.

12. The method of claim 10, wherein:

the adjusted sampling characteristics include characteristics corresponding to inputting less than the determined nominal amount of the full input data set.

13. The method of claim 10, wherein:

the determined characteristic associated with the determined amount of data includes at least one of the group consisting of

statistical confidence for analysis that would be achieved if carried out based on the determined amount of data;

an amount of time to import the determined amount of data; and

a percentage of the full input data set of which the determined amount of data comprises.

14. A computing device comprising processing circuitry and memory circuitry, wherein the computing device is configured to import an amount of data of a full input data set that is a portion of the full input data set, wherein the full input data set includes data records of a dimensionally-modeled fact collection, wherein the portion of the full input data set to import has been determined by:

determining an amount of the data of the full input set to import based on an amount of available memory of the computer system;

based on the determined amount of the data to import and on characteristics of the full input data set at least other than the total size of the full input data set, determining sampling characteristics for sampling the full input data set; and

sampling the full input data set, to determine the portion of the records to import, in accordance with the determined sampling characteristics, wherein the sampling characteristics are determined such that analysis as a result of processing by the executing application of the sampled portion of the records imported is representative of the analysis that could otherwise be carried out on the full input data set, with a calculable statistical relevance.

15. The computing device of claim 14, wherein:

the computing device is further configured to process the portion of the full input data set.

16. The computing device of claim 14, wherein:

the computing device is further configured to cause a representation of the portion of the full input data set to be displayed.

17. A computer program product for importing a portion of data records of a full input data set into memory of a computer system for processing by an executing application, wherein the full input data set includes data records of a dimensionally-modeled fact collection, the computer program product comprising at least one computer-readable medium having computer program instructions stored therein which are operable to cause at least one computing device to:

determine an amount of the data of the full input set to import based on an amount of available memory of the computer system;

based on the determined amount of the data to import and on characteristics of the full input data set at least other than the total size of the full input data set, determine sampling characteristics for sampling the full input data set; and

cause a portion of the records of the full input data set to be imported into the memory of the computer system, including sampling the full input data set, to determine the portion of the records to import, in accordance with the determined sampling characteristics, wherein the sampling characteristics are determined such that analysis as a result of processing by the executing application of the sampled portion of the records imported is representative of the analysis that could otherwise be carried out on the full input data set, with a calculable statistical relevance.

18. The computer program product of claim 17, wherein:

the computer program instructions operable to cause the at least one computing device to sample the full input data set, to determine the portion of the records to import includes computer program instructions operable to cause the at least one computing device to sample such that, for a particular dimension of the fact collection, for each record of the full input data set sampled, wherein that input record has a particular value at the particular dimension, every record of the full input data set having that particular value at the particular dimension is sampled.

19. The computer program product of claim 17, wherein:

the computer program instructions operable to cause the at least one computing device to sample the full input data set, to determine the portion of the records to import includes computer program instructions operable to cause the at least one computing device to apply a Fowler/Noll/Vo (FNV) Hash algorithm for sampling based on a key that is the value for each record at a particular dimension.

20. The computer program product of claim 17, wherein:

each data record includes information regarding behavior of a particular user with respect to online services.

21. The computer program product of claim 20, wherein:

the computer program instructions operable to cause the at least one computing device to sample the full input data set, to determine the portion of the records to import includes computer program instructions operable to sample such that, for each record of the full input data set sampled, wherein that input record has information about a particular user, every record of the full input data set including information regarding behavior of the particular user is sampled.

22. The computer program product of claim 21, wherein the computer program instructions further include computer program instructions operable to cause the at least one computing device to:

analyze the sampled data records, including determining a statistical confidence for the analysis; and

cause a result of the analysis, including the determined statistical confidence, to be made available to a user.

23. The computer program product of claim 17, wherein the computer program instructions operable to cause the at least one computing device to determine an amount of the data of the full input set to import based on an amount of available memory of the computer system includes computer program instructions operable to cause the at least one computing device to:

determine the amount of the data to be less than the amount of available memory of the computer system.

24. The computer program product of claim 17, wherein the computer program instructions operable to cause the at least one computing device to sample the full input data set, to determine the portion of the records to import includes computer program instructions operable to cause the at least one computing device to:

assign the full input data set into a plurality of bins, wherein for each record of the full input data set sampled, that input record has a particular value at a particular dimension and every record of the full input data set having the particular value at the particular dimension is in that same bin; and

sample all the records of at least one of the bins.