System and method for data aggregation and analysis

• US 10,146,954 B1
• Filed: 07/31/2017
• Issued: 12/04/2018
• Est. Priority Date: 06/11/2012
• Status: Active Grant

• Current Assignee: Quest Software, Inc.
• Original Assignee: Quest Software, Inc.
• Inventors: Brisebois, Michel Albert, Aylesworth, Jason, Johnstone, Curtis T., Leach, Andrew John, Vinogradov, Elena V., Blaiberg, Joel Stacy, Pope, Stephen, Holmesdale, Shawn Donald, Hu, GuangNing
• Primary Examiner(s): Parsons, Theodore C
• Application Number:
  US15/664,849
• Time in Patent Office: 491 Days
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/6227 where protection concerns the structure of data, e.g. records, types, queries
• CPC Class Codes: G06F 2216/01 Automatic library building
• CPC Class Codes: G06F 2216/03 Data mining
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

In one embodiment, a method includes managing and controlling a plurality of data-access credentials. The method further includes accessing data from a plurality, of sources in a plurality of data formats. The accessing includes using one or more data-access credentials of the plurality of data-access credentials. The one or more data-access credentials are associated with at least a portion of the plurality of data sources. The method also includes abstracting the data into a standardized format for further analysis. The abstracting includes selecting the standardized format based on a type of the data. In addition, the method includes applying a security policy to the data. The applying includes identifying at least a portion of the data for exclusion from storage based on the security policy. Additionally, the method includes filtering from storage any data identified for exclusion. Further, the method includes storing the data in the standardized format.

Claims

1. A method comprising:

on a computer system comprising at least one server computer, managing and controlling a plurality of data-access credentials;

accessing, by the computer system, data from a plurality of sources in a plurality of data formats, the plurality of sources comprising sources that are internal to the computer system and sources that are external to the computer system;

wherein the accessing comprises using one or more data-access credentials of the plurality of data-access credentials, the one or more data-access credentials being associated with at least a portion of the plurality of data sources;

abstracting, by the computer system, the data into a standardized format for further analysis, the abstracting comprising selecting the standardized format based on a type of the data;

applying, by the computer the system, a security policy to the data;

wherein the applying comprises identifying at least a portion of the data for exclusion from storage based on the security policy;

the computer system filtering from storage any data identified for exclusion;

classifying the data based on one or more characteristics of metadata associated with the data;

wherein the classifying comprises performing one or more probabilistic algorithms to determine a probability that a set of data comprises a particular classification based on a combination of probabilistic determinations associated with subsets of the set of data, parameters and metadata associated with the set of data;

storing, by the computer system, the filtered data in the standardized format;

modifying the one or more probabilistic algorithms based, at least in part, on a determined accuracy of previously classified data; and

responsive to the modification, reclassifying the classified data using the modified one or more probabilistic algorithms.

2. The method of claim 1, wherein the using comprises using the one or more data-access credentials to access an account for each of the at least a portion of the plurality of data sources.

3. The method of claim 1, wherein the accessing comprises accessing metadata associated with the data.

4. The method of claim 1, wherein the abstracting comprises extracting metadata from the data.

5. The method of claim 1, wherein the applying comprises preventing the computer system from accessing the at least a portion of the data for exclusion.

6. The method of claim 1, comprising, prior to the storing:

formatting the data for storage.

7. The method of claim 1, wherein the storing comprises dynamically expanding a database responsive to the data including a new type of data.

8. The method of claim 1, wherein the plurality of sources are selected from the group consisting of:

cloud services, social-media services, hosted applications, and email servers.

9. The method of claim 1, wherein:

the data comprises messaging data; and

the standardized format comprises a standardized format for messages.

10. The method of claim 9, wherein the accessing comprises accessing email from an email server to obtain metadata.

11. An information handling system comprising:

a computer processor and memory, wherein the computer processor and memory is operable to implement a method comprising;

managing and controlling a plurality of data-access credentials;

accessing data from a plurality of sources in a plurality of data formats, the plurality of sources comprising sources that are internal to the information handling system and sources that are external to the information handling system;

wherein the accessing comprises using one or more data-access credentials of the plurality of data-access credentials, the one or more data-access credentials being associated with at least a portion of the plurality of data sources;

abstracting the data into a standardized format for further analysis, the abstracting comprising selecting the standardized format based on a type of the data;

applying a security policy to the data;

wherein the applying comprises identifying at least a portion of the data for exclusion from storage based on the security policy;

filtering from storage any data identified for exclusion;

classifying the data based on one or more characteristics of metadata associated with the data;

wherein the classifying comprises performing one or more probabilistic algorithms to determine a probability that a set of data comprises a particular classification based on a combination of probabilistic determinations associated with subsets of the set of data, parameters and metadata associated with the set of data;

storing the filtered data in the standardized format;

modifying the one or more probabilistic algorithms based, at least in part, on a determined accuracy of previously classified data; and

responsive to the modification, reclassifying the classified data using the modified one or more probabilistic algorithms.

12. The information handling system of claim 11, wherein the using comprises using the one or more data-access credentials to access an account for each of the at least a portion of the plurality of data sources.

13. The information handling system of claim 11, wherein the accessing comprises accessing metadata associated with the data.

14. The information handling system of claim 11, wherein the abstracting comprises extracting metadata from the data.

15. The information handling system of claim 11, wherein the applying comprises preventing the computer system from accessing the at least a portion of the data for exclusion.

16. The information handling system of claim 11, comprising, prior to the storing:

formatting the data for storage.

17. The information handling system of claim 11, wherein the storing comprises dynamically expanding a database responsive to the data including a new type of data.

18. The information handling system of claim 11, wherein the plurality of sources are selected from the group consisting of:

cloud services, social-media services, hosted applications, and email servers.

19. The information handling system of claim 11, wherein:

the data comprises messaging data; and

the standardized format comprises a standardized format for messages.

20. A computer-program product comprising a non-transitory computer-usable medium having computer-readable program code embodied therein, the computer-readable program code adapted to be executed to implement a method comprising:

on a computer system comprising at least one server computer, managing and controlling a plurality of data-access credentials;

accessing, by the computer system, data from a plurality of sources in a plurality of data formats, the plurality of sources comprising sources that are internal to the computer system and sources that are external to the computer system;

wherein the accessing comprises using one or more data-access credentials of the plurality of data-access credentials, the one or more data-access credentials being associated with at least a portion of the plurality of data sources;

abstracting, by the computer system, the data into a standardized format for further analysis, the abstracting comprising selecting the standardized format based on a type of the data;

applying, by the computer the system, a security policy to the data;

wherein the applying comprises identifying at least a portion of the data for exclusion from storage based on the security policy;

the computer system filtering from storage any data identified for exclusion;

classifying the data based on one or more characteristics of metadata associated with the data;

wherein the classifying comprises performing one or more probabilistic algorithms to determine a probability that a set of data comprises a particular classification based on a combination of probabilistic determinations associated with subsets of the set of data, parameters and metadata associated with the set of data;

storing, by the computer system, the filtered data in the standardized format;

modifying the one or more probabilistic algorithms based, at least in part, on a determined accuracy of previously classified data; and

responsive to the modification, reclassifying the classified data using the modified one or more probabilistic algorithms.