Detection and quantifying of data redundancy in column-oriented in-memory databases

• US 9,785,660 B2
• Filed: 09/25/2014
• Issued: 10/10/2017
• Est. Priority Date: 09/25/2014
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Said, Bare, Jentsch, Frank
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US14/496,715
• Publication Number:
  US 20160092477A1
• Time in Patent Office: 1,111 Days
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing

Abstract

Methods, systems, and computer-readable storage media for quantifying a redundancy of data stored in tables of a database. In some implementations, actions include, for each primary key and table pair in a set of primary key and table pairs, determining an aggregate severity sub-score based on one or more values of the primary key in the table, the primary key being included in a set of primary keys and the table being included in a set of tables, determining an aggregate severity score for each primary key in the set of primary keys based on aggregate severity sub-scores associated with the primary key to provide a plurality of aggregate severity scores, each aggregate severity score indicating a relative redundancy of values of the primary key across all tables in the set of tables, and providing a list of aggregate severity scores and corresponding primary keys for display to a user.

Claims

1. A computer-implemented method for quantifying a redundancy of data stored in tables of a column-oriented in-memory database, the method being executed using one or more processors and comprising:

determining, by the one or more processors, data structures for a plurality of data objects stored in the column-oriented in-memory database through an integration of distributed enterprise systems, each data structure defining one or more attributes, one attribute being a primary key of a respective data object;

for each primary key and table pair in a set of primary key and table pairs, determining, by the one or more processors, an aggregate severity sub-score based on one or more values of the primary key in the table, the primary key being included in a set of primary keys and the table being included in a set of tables;

determining, by the one or more processors, an aggregate severity score for each primary key in the set of primary keys based on aggregate severity sub-scores associated with the primary key to provide a plurality of aggregate severity scores, each aggregate severity score indicating a relative redundancy of values of the primary key across all tables in the set of tables and each severity sub-score being determined based on a number of occurrences of the values of the primary key across all the tables and an attribute weight that is inversely proportional to an association grade for the primary key relative to the primary table;

providing, by the one or more processors, a list of aggregate severity scores and corresponding primary keys for display to a user; and

performing, by the one or more processors, at least one operation to reduce the relative redundancy associated with the primary key of the set of primary keys based on the list of aggregate severity scores.

2. The method of claim 1, wherein determining an aggregate severity sub-score based on one or more values of the primary key in the table for a primary key and table pair comprises:

determining a value severity score based on the plurality of value severity sub-scores; and

determining the aggregate severity sub-score based on a plurality of value severity scores, the value severity score being included in the plurality of value severity scores.

3. The method of claim 2, wherein the aggregate severity sub-score is further based on a number of entries of the table of the primary key and table pair.

4. The method of claim 1, further comprising obtaining a set of associated attributes for each primary key in the set of primary keys based on a chain of tables in the set of tables, the chain of tables comprising two or more tables.

5. The method of claim 4, further comprising, for each associated attribute in the set of associated attributes, determining an attribute weight based on a degree of indirection in the chain of tables.

6. The method of claim 1, wherein the list of aggregate severity scores is provided as a ranked list of scores based on respective values of the plurality of aggregate severity scores.

7. The method of claim 1, wherein the primary key of the set of primary keys is associated with the highest aggregate severity score within the set of primary keys.

8. A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for quantifying a redundancy of data stored in tables of a column-oriented in-memory database, the operations comprising:

determining data structures for a plurality of data objects stored in the column-oriented in-memory database through an integration of distributed enterprise systems, each data structure defining one or more attributes, one attribute being a primary key of a respective data object;

for each primary key and table pair in a set of primary key and table pairs, determining an aggregate severity sub-score based on one or more values of the primary key in the table, the primary key being included in a set of primary keys and the table being included in a set of tables;

determining an aggregate severity score for each primary key in the set of primary keys based on aggregate severity sub-scores associated with the primary key to provide a plurality of aggregate severity scores, each aggregate severity score indicating a relative redundancy of values of the primary key across all tables in the set of tables and each severity sub-score being determined based on a number of occurrences of the values of the primary key across all the tables and an attribute weight that is inversely proportional to an association grade for the primary key relative to the primary table;

providing a list of aggregate severity scores and corresponding primary keys for display to a user; and

performing at least one operation to reduce the relative redundancy associated with the primary key of the set of primary keys based on the list of aggregate severity scores.

9. The computer-readable storage medium of claim 8, wherein determining an aggregate severity sub-score based on one or more values of the primary key in the table for a primary key and table pair comprises:

determining a value severity score based on the plurality of value severity sub-scores; and

determining the aggregate severity sub-score based on a plurality of value severity scores, the value severity score being included in the plurality of value severity scores.

10. The computer-readable storage medium of claim 9, wherein the aggregate severity sub-score is further based on a number of entries of the table of the primary key and table pair.

11. The computer-readable storage medium of claim 8, wherein operations further comprise obtaining a set of associated attributes for each primary key in the set of primary keys based on a chain of tables in the set of tables, the chain of tables comprising two or more tables.

12. The computer-readable storage medium of claim 11, wherein operations further comprise, for each associated attribute in the set of associated attributes, determining an attribute weight based on a degree of indirection in the chain of tables.

13. The computer-readable storage medium of claim 8, wherein the list of aggregate severity scores is provided as a ranked list of scores based on respective values of the plurality of aggregate severity scores.

14. The computer-readable storage medium of claim 8, wherein operations further comprise:

receiving user input based on the list of aggregate severity scores, the user input indicating a command to execute an operation to reduce redundancy associated with a primary key of the set of primary keys; and

performing the operation.

15. A system, comprising:

a computing device; and

a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for quantifying a redundancy of data stored in tables of a column-oriented in-memory database, the operations comprising;

determining data structures for a plurality of data objects stored in the column-oriented in-memory database through an integration of distributed enterprise systems, each data structure defining one or more attributes, one attribute being a primary key of a respective data object;

for each primary key and table pair in a set of primary key and table pairs, determining an aggregate severity sub-score based on one or more values of the primary key in the table, the primary key being included in a set of primary keys and the table being included in a set of tables;

determining an aggregate severity score for each primary key in the set of primary keys based on aggregate severity sub-scores associated with the primary key to provide a plurality of aggregate severity scores, each aggregate severity score indicating a relative redundancy of values of the primary key across all tables in the set of tables and each severity sub-score being determined based on a number of occurrences of the values of the primary key across all the tables and an attribute weight that is inversely proportional to an association grade for the primary key relative to the primary table;

providing a list of aggregate severity scores and corresponding primary keys for display to a user; and

performing at least one operation to reduce the relative redundancy associated with the primary key of the set of primary keys based on the list of aggregate severity scores.

16. The system of claim 15, wherein determining an aggregate severity sub-score based on one or more values of the primary key in the table for a primary key and table pair comprises:

determining a value severity score based on the plurality of value severity sub-scores; and

determining the aggregate severity sub-score based on a plurality of value severity scores, the value severity score being included in the plurality of value severity scores.

17. The system of claim 16, wherein the aggregate severity sub-score is further based on a number of entries of the table of the primary key and table pair.

18. The system of claim 15, wherein operations further comprise obtaining a set of associated attributes for each primary key in the set of primary keys based on a chain of tables in the set of tables, the chain of tables comprising two or more tables.

19. The system of claim 18, wherein operations further comprise, for each associated attribute in the set of associated attributes, determining an attribute weight based on a degree of indirection in the chain of tables.

20. The system of claim 15, wherein the list of aggregate severity scores is provided as a ranked list of scores based on respective values of the plurality of aggregate severity scores.