TECHNIQUES FOR APPLICATION DATA SCRUBBING, REPORTING, AND ANALYSIS

• US 20090240694A1
• Filed: 03/18/2008
• Published: 09/24/2009
• Est. Priority Date: 03/18/2008
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: Novell Intellectual Property Holdings, Inc.
• Inventors: Morgan, Charles Matthew, Jensen, Nathan Blaine, Carter, Stephen R., Scheuber-Heinz, Volker Gunnar
• Granted Patent: US 8,838,652 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor

Abstract

Techniques for application data scrubbing, reporting, and analysis are presented. A plurality of data sources are analyzed in accordance with their schemas and matching rules. Merging rules are applied to merge a number of data types across the data sources together. A report is produced for inspection and a master data source is generated. The processing can be iterated with rules modified in response to the report for purposes of refining the master data source.

Claims

1. A machine-implemented method, comprising:

acquiring a first schema for a first data source and a second schema for a second data source;

using the first and second schemas to detect data types and patterns for the data types in both the data sources;

matching some first patterns associated with the first data source to other second patterns associated with the second data source in response to matching rules; and

generating a report that identifies the matched first patterns of the first data source to the second patterns of the second source.

2. The method of claim 1 further comprising, merging selective ones of the first patterns with selective ones of the second patterns to produce a master data source in response to the report and merge rules.

3. The method of claim 2 further comprising, permitting a data analyst to interact with and modify the report by altering the merge rules or adding new merge rules and then reiterating the processing of the method after the data analyst modifies the report to produce a modified master data source.

4. The method of claim 1, wherein matching further includes applying a policy against data in the first and second data sources to generate statistics to use with the matching rules.

5. The method of claim 1 further comprising, producing a duplicate report that identifies selective first patterns from the first data source that are duplicated in selective second patterns from the second data source.

6. The method of claim 5, further comprising using the duplicate report to retain a single version of the duplicated patterns in a modified version of a master data source.

7. The method of claim 1, further comprising generating white and black list reports for a master data source, wherein the white list report identifies data from the first and second data sources that conforms to policy, and wherein the black list report identifies other data from the first and second data sources that do not conform to the policy and that are to be cleaned or edited for correction to conform to the policy.

8. A machine-implemented method, comprising:

interacting with a data analyst via an interface presented to the data analyst;

receiving identifiers for data schemas and data sources associated with those data schemas from the data analyst via the interface;

acquiring merge rules from the data analyst via the interface, wherein the merge rules identify conditions within the data sources for merging different data types defined in the data schemas together with one another; and

parsing the data sources using the data schemas and enforcing the merge rules to produce a merge report and to produce a master data source that combines the data sources together in accordance with the merge rules.

9. The method of claim 8 further comprising, receiving modified merge rules from the data analyst via the interface that identify modified conditions with the data sources for merging the different data types and re-parsing the data sources to produce a modified master data source.

10. The method of claim 8 further comprising, comparing the merge report to one or more previously generated merge reports to profile changes in data types for the data sources defined in the data schemas over time.

11. The method of claim 8, wherein parsing further includes identifying matching rules from the data schemas, wherein the matching rules assist in identifying data types and patterns in the data sources during the parsing.

12. The method of claim 8, wherein parsing further includes acquiring from the data analyst via the interface one or more additional matching rules.

13. The method of claim 8 further comprising, generating a duplicate report that identifies duplicate data types across different ones of the data sources.

14. The method of claim 8 further comprising, generating a black list report that identifies data types from the data sources that are to be corrected by manual or automated mechanisms.

15. A machine-implemented system, comprising:

a data analysis tool implemented in a machine-accessible and computer-readable medium and to execute on a machine; and

a data analyzer implemented in a machine-accessible and computer-readable medium and to execute on the machine or another machine of a network;

wherein the data analysis tool is adapted to provide an interface to a data analyst that permits the data analyst to identify data sources for analysis, and wherein the data analyzer is to acquire a separate data schema for each of the data sources and uses the data schemas to parse the data sources to identify data types and patterns, and wherein the data analyzer uses merge rules and policies to merge some of the data types and their corresponding data from the data sources together in a master data source.

16. The system of claim 15, wherein the data sources are one or more of the following:

a directory, a relational database table, a file, a World-Wide Web (WWW) page, and output produced from an application that processes on a machine of the network.

17. The system of claim 15, wherein the data analyzer iterates over the data sources a configurable number of times in response to direction received from the data analyst via the data analyst tool to produced modified versions of the master data source.

18. The system of claim 17, wherein the data analyst modifies one or more of the merge rules or policies during at least one iteration.

19. The system of claim 15, wherein the data analysis tool generates a merge report that includes statistics regarding actions taking when the data types and the patterns are identified and that includes information regarding the merge rules and the policies applied.

20. The system of claim 15, wherein the data analysis tool generates a duplicate report that identifies duplicate data types that span two or more of the data sources.

21. A machine-implemented system, comprising:

applications implemented in a machine-accessible and computer-readable medium and to process on multiple machines of a network; and

a data analyzer tool implemented in a machine-accessible and computer-readable medium and to process on a machine of the network;

wherein each application produces application data defined by its own schema and wherein the data analyzer parses the application data using the schemas and further uses merge rules and policies to map the application data to a master data source.

22. The system of claim 21, wherein the data analyzer iterates the application data a configurable number of times in response to modified merge rules and modified policies.

23. The system of claim 21, wherein the data analyzer generates a merge report, a duplicate data report, a white list report, and a black list report.

24. The system of claim 21, wherein the data analyzer also uses a master schema to assist in mapping to the application data to the master data source.

25. The system of claim 21, wherein the master data source acquires a unique identity from an identity manager that processing on a machine of the network, and wherein security restrictions are enforced against the master data source via the identity manager.