Dynamic visual profiling and visualization of high volume datasets and real-time smart sampling and statistical profiling of extremely large datasets

• US 10,296,192 B2
• Filed: 09/24/2015
• Issued: 05/21/2019
• Est. Priority Date: 09/26/2014
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Stojanovic, Alexander Sasha, Rivas, Luis E., Markey, Kevin L., Bidwell, Christopher F.
• Primary Examiner(s): Casillas, Roland J
• Application Number:
  US14/864,505
• Publication Number:
  US 20160092090A1
• Time in Patent Office: 1,335 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

The present disclosure relates generally to a data enrichment service that automatically profiles data sets and provides visualizations of the profiles using a visual-interactive model within a client application (such as a web browser or mobile app). The visual profiling can be refined through end user interaction with the visualization objects and guide exploratory data visualization and discovery. Additionally, data sampling of heterogeneous data streams can be performed during ingestion to extract statistical attributes from multi-columnar data (e.g., standard deviation, median, mode, correlation coefficient, histogram, etc.). Data sampling can continue in real-time as data sources are updated.

Claims

1. A method comprising:

analyzing, by a computer system of a data enrichment service, one or more columns of a data set obtained from one or more data sources;

identifying, by the computer system of the data enrichment service, one or more patterns in the one or more columns of the data set obtained from one or more data sources, wherein the one or more patterns comprises a repeated type of data;

matching the one or more identified patterns to entity information from a knowledge system;

identifying a category to which the one or more patterns in the one or more columns of the data set belongs;

generating a graphical visualization based on the one or more identified categories of the one or more identified patterns matching the entity information;

causing the graphical visualization to be displayed in a user interface;

receiving input related to the graphical visualization through the user interface; and

updating the graphical visualization based on the input.

2. The method of claim 1, further comprising:

monitoring the one or more data sources in real-time;

periodically updating the identified one or more patterns based on updated data obtained from the one or more data sources by the monitoring;

generating an updated graphical visualization based on an update to the identified one or more patterns; and

causing the updated graphical visualization to be displayed in the user interface.

3. The method of claim 1, wherein identifying the one or more patterns in the data includes:

profiling the data as it is obtained from the one or more data sources in order to determine the one or more patterns in the data set obtained from the one or more data sources; and

determining sampled data that is a representative sample of one or more characteristics within the profiled data, wherein determining the sampled data includes oversampling a first portion of the profiled data and undersampling a second portion of the profiled data.

4. The method of claim 3, wherein the graphical visualization includes a graphical dashboard, and wherein the graphical dashboard indicates a plurality of metrics, each of the plurality of metrics indicating a real time metric of the data relative to a time that the data is profiled.

5. The method of claim 4, wherein the plurality of metrics represent a user-specific set of metrics calculated for the data.

6. The method of claim 4, wherein receiving the input related to the graphical visualization through the user interface includes:

receiving a selection of a first point in time in the user interface;

displaying a first plurality of metrics associated with the first point in time;

receiving a gesture-based input indicating a second point in time in the user interface; and

displaying a second plurality of metrics associated with the second point in time.

7. The method of claim 4, wherein receiving the input related to the graphical visualization through the user interface includes:

receiving a gesture-based input in the graphical dashboard; and

based on the gesture-based input, adjusting a displayed time scale of the graphical dashboard.

21. The method according to claim 1, wherein the category comprises one of an address, a business name, a location, a name of a person, an identification number, a date, a time, a domain addresses, an email address, a social security number, a zip code, a phone number, and a credit card number.

22. The method according to claim 1, wherein the one or more patterns comprises a repeated string of data.

8. A system comprising:

a plurality of data sources;

a plurality of data targets; and

a cloud computing infrastructure system comprising;

one or more processors communicatively coupled to the plurality of data sources and communicatively coupled to the plurality of data targets, over at least one communication network; and

a memory coupled to the one or more processors, the memory storing instructions to provide a data enrichment service, wherein the instructions, when executed by the one or more processors, cause the one or more processors to;

analyze one or more columns of a data set obtained from one or more data sources;

identify one or more patterns in the one or more columns of the data set obtained from one or more data sources, wherein the one or more patterns comprises a repeated type of data;

match the one or more identified patterns to entity information from a knowledge system;

identify a category to which the one or more patterns in the one or more columns of the data set belongs;

generate a graphical visualization based on the one or more identified categories of the one or more identified patterns matching the entity information;

cause the graphical visualization to be displayed in a user interface;

receive input related to the graphical visualization through the user interface; and

update the graphical visualization based on the input.

9. The system of claim 8, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

monitor the one or more data sources in real-time;

periodically update the identified pattern based on updated data obtained from the one or more data sources by the monitoring;

generate an updated graphical visualization based on an update to the identified one or more patterns; and

cause the updated graphical visualization to be displayed in the user interface.

10. The system of claim 8, wherein identifying the one or more patterns in the data includes:

profiling the data as it is obtained from the one or more data sources in order to determine the one or more patterns in the plurality of data sources obtained from the one or more data sources; and

determining sampled data that is a representative sample of one or more characteristics within the profiled data, wherein determining the sampled data includes oversampling a first portion of the profiled data and undersampling a second portion of the profiled data.

11. The system of claim 8, wherein the graphical visualization includes a graphical dashboard, and wherein the graphical dashboard indicates a plurality of metrics, each of the plurality of metrics indicating a real time metric of the data relative to a time that the data is profiled.

12. The system of claim 11, wherein the plurality of metrics represent a user-specific set of metrics calculated for the data.

13. The system of claim 11, wherein receiving the input related to the graphical visualization through the user interface includes:

receiving a selection of a first point in time in the user interface;

displaying a first plurality of metrics associated with the first point in time;

receiving a gesture-based input indicating a second point in time in the user interface; and

displaying a second plurality of metrics associated with the second point in time.

14. The system of claim 11, wherein receiving the input related to the graphical visualization through the user interface includes:

receiving a gesture-based input in the graphical dashboard; and

based on the gesture-based input, adjusting a displayed time scale of the graphical dashboard.

15. A non-transitory computer readable storage medium including instructions stored thereon which, when executed by one or more processors, cause the one or more processors to:

analyze, by a computer system of a data enrichment service, one or more columns of a data set obtained from one or more data sources;

identify, by the computer system of the data enrichment service, one or more patterns in the one or more columns of the data set obtained from one or more data sources, wherein the one or more patterns comprises a repeated type of data;

match the one or more identified patterns to entity information from a knowledge system;

identify a category to which the one or more patterns in the one or more columns of the data set belongs;

generate a graphical visualization based on the one or more identified categories of the one or more identified patterns matching the entity information;

cause the graphical visualization to be displayed in a user interface;

receive input related to the graphical visualization through the user interface; and

update the graphical visualization based on the input.

16. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

monitor the one or more data sources in real-time;

periodically update the identified pattern based on updated data obtained from the one or more data sources by the monitoring;

generate an updated graphical visualization based on an update to the identified one or more patterns; and

cause the updated graphical visualization to be displayed in the user interface.

17. The non-transitory computer readable storage medium of claim 15, wherein identifying the one or more patterns in the data includes:

profiling the data as it is obtained from the one or more data sources in order to determine the one or more patterns in the data set obtained from the one or more data sources; and

determining sampled data that is a representative sample of one or more characteristics within the profiled data, wherein determining the sampled data includes oversampling a first portion of the profiled data and undersampling a second portion of the profiled data.

18. The non-transitory computer readable storage medium of claim 17, wherein the graphical visualization includes a data visualization dashboard, the data visualization dashboard including a plurality of metric indicators, wherein each of the plurality of metric indicators indicate a real time metric of profile the data over a time period, and wherein the plurality of metrics indicators include a user-selected subset of global metrics calculated for the data.

19. The non-transitory computer readable storage medium of claim 18, wherein receiving the input related to the graphical visualization through the user interface includes:

receiving a selection of a first point in time in the user interface;

displaying a first plurality of metrics associated with the first point in time;

receiving a gesture-based input indicating a second point in time in the user interface; and

displaying second plurality of metrics associated with the second point in time.

20. The non-transitory computer readable storage medium of claim 18, wherein receiving input related to the graphical visualization through the user interface includes:

receiving a gesture-based input in the data visualization dashboard; and

based on the gesture-based input, adjusting a displayed time scale of the data visualization dashboard.