Database operation using metadata of data sources

• US 10,579,627 B2
• Filed: 01/08/2016
• Issued: 03/03/2020
• Est. Priority Date: 01/08/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nevrekar, Narayan, Stugren, Francisc, Bhartia, Atul, Fischer, Kevin Joseph, Guthrie, Michael, Mohammed, Sohail, Rosemund, Scott, Al-Rashid, Kazi
• Primary Examiner(s): Morrison, Jay A
• Assistant Examiner(s): Caiado, Antonio J
• Application Number:
  US14/991,388
• Publication Number:
  US 20170199875A1
• Time in Patent Office: 1,516 Days
• Field of Search: 707603
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9032 Query formulation
• CPC Class Codes: G06F 40/186 Templates

Abstract

In some examples, a computing device determines a data manipulation from a job specification. The device determines a corresponding data-processing instruction using data-source metadata, and determines and executes a corresponding query. In some examples, a device receives search keys. The device searches data-source metadata using the search keys. The device weights a first data source based on producer-consumer relationships between data sources, and ranks the first data source using the weight. In some examples, a device determines structural and content information of a data record. The device determines a data-source identifier from the structural information and stores the content information with the data-source identifier in a database. In some examples, via a user interface, a device receives a job specification and annotation data. The device stores the spec and the annotation data in a metadata repository.

Claims

1. A system comprising:

computer-readable media having thereon a plurality of modules; and

at least one processing unit operably coupled with the computer-readable media, configured to access data from one or more data sources, and adapted to execute one or more modules of the plurality of modules comprising;

at least one module of a construction engine configured to;

determine a dataflow corresponding to a job specification associated with one or more data sources;

determine a data manipulation to operate on the job specification based on the dataflow;

determine a data-processing instruction to perform functions of the data manipulation based on the data manipulation and at least some metadata of the one or more data sources; and

determine a query by assembling the data-processing instruction based at least in part on the job specification;

at least one module of an execution engine configured to;

execute the query by accessing the data from at least one of the one or more data sources to provide query results; and

at least one module of an ingestion engine configured to;

retrieve a data record;

determine metadata corresponding to the data record by extracting attribute values or relationships from the data, the metadata including a data-source identifier indicating at least one of the one or more data sources; and

store the metadata in a metadata repository in association with the data-source identifier.

2. The system as claim 1 recites, wherein the construction engine is further configured to:

determine a plurality of data manipulations, the plurality of data manipulations including the data manipulation;

determine a plurality of data-processing instructions based at least in part on respective data manipulations of the plurality of data manipulations, the plurality of data-processing instructions including the data-processing instruction;

determine a relative order of at least two data manipulations of the plurality of data manipulations, wherein at least two data-processing instructions correspond to the at least two data manipulations, respectively; and

determine the query further by assembling the at least two data-processing instructions according to the relative order.

3. The system as claim 1 recites, wherein the construction engine is further configured to determine the data-processing instruction by:

retrieving a stored template corresponding to the data manipulation; and

populating at least one field in the stored template based at least in part on the at least some metadata of the one or more data sources to provide the data-processing instruction.

4. The system as claim 1 recites, the plurality of modules further comprising one or more modules of a reporting engine configured to:

determine an output format based at least in part on the job specification;

determine a second data-processing instruction to perform at least one of a copy, a move, or a transformation of the query results according to the output format; and

execute the second data-processing instruction to provide reported data corresponding to the query results based at least in part on the output format.

5. The system as claim 1 recites, the plurality of modules further comprising at least one module configured to:

provide a user interface comprising a data-source selection control, a query-criterion control, and a result-field control; and

receive the job specification via the user interface.

6. A computer-implemented method, comprising:

determining a dataflow corresponding to a job specification associated with one or more data sources;

determining a data manipulation to operate on the job specification based on the dataflow;

determining a data-processing instruction to perform functions of the data manipulation based on the data manipulation and at least some metadata of the one or more data sources; and

determining a query by assembling the data-processing instruction based at least in part on the job specification;

executing the query by accessing data from at least one of the one or more data sources to provide query results; and

retrieving a data record;

determining metadata corresponding to the data record by extracting attribute values or relationships from the data, the metadata including a data-source identifier indicating at least one of the one or more data sources; and

storing the metadata in a metadata repository in association with the data-source identifier.

7. The computer-implemented method of claim 6, further comprising:

determining a plurality of data manipulations, the plurality of data manipulations including the data manipulation;

determining a plurality of data-processing instructions based at least in part on respective data manipulations of the plurality of data manipulations, the plurality of data-processing instructions including the data-processing instruction;

determining a relative order of at least two data manipulations of the plurality of data manipulations, wherein at least two data-processing instructions correspond to the at least two data manipulations, respectively; and

determining the query further by assembling the at least two data-processing instructions according to the relative order.

8. The computer-implemented method of claim 6, wherein the determining the data-processing instruction comprises:

retrieving a stored template corresponding to the data manipulation; and

populating at least one field in the stored template based at least in part on the at least some metadata of the one or more data sources to provide the data-processing instruction.

9. The computer-implemented method of claim 6, further comprising:

determining an output format based at least in part on the job specification;

determining a second data-processing instruction to perform at least one of a copy, a move, or a transformation of the query results according to the output format; and

executing the second data-processing instruction to provide reported data corresponding to the query results based at least in part on the output format.

10. The computer-implemented method of claim 6, further comprising:

providing a user interface comprising a data-source selection control, a query-criterion control, and a result-field control; and

receiving the job specification via the user interface.

11. A non-transitory computer-readable medium comprising instructions executable by a processor, comprising:

instructions to determine a dataflow corresponding to a job specification associated with one or more data sources;

instructions to determine a data manipulation to operate on the job specification based on the dataflow;

instructions to determine a data-processing instruction to perform functions of the data manipulation based on the data manipulation and at least some metadata of the one or more data sources; and

instructions to determine a query by assembling the data-processing instruction based at least in part on the job specification;

instructions to execute the query by accessing data from at least one of the one or more data sources to provide query results; and

instructions to retrieve a data record;

instructions to determine metadata corresponding to the data record by extracting attribute values or relationships from the data, the metadata including a data-source identifier indicating at least one of the one or more data sources; and

instructions to store the metadata in a metadata repository in association with the data-source identifier.

12. The non-transitory computer-readable medium of claim 11, further comprising:

instructions to determine a plurality of data manipulations, the plurality of data manipulations including the data manipulation;

instructions to determine a plurality of data-processing instructions based at least in part on respective data manipulations of the plurality of data manipulations, the plurality of data-processing instructions including the data-processing instruction;

instructions to determine a relative order of at least two data manipulations of the plurality of data manipulations, wherein at least two data-processing instructions correspond to the at least two data manipulations, respectively; and

instructions to determine the query further by assembling the at least two data-processing instructions according to the relative order.

13. The non-transitory computer-readable medium of claim 11, further comprising:

instructions to retrieve a stored template corresponding to the data manipulation; and

instructions to populate at least one field in the stored template based at least in part on the at least some metadata of the one or more data sources to provide the data-processing instruction.

14. The non-transitory computer-readable medium of claim 11, further comprising:

instructions to determine an output format based at least in part on the job specification;

instructions to determine a second data-processing instruction to perform at least one of a copy, a move, or a transformation of the query results according to the output format; and

instructions to execute the second data-processing instruction to provide reported data corresponding to the query results based at least in part on the output format.

15. The non-transitory computer-readable medium of claim 11, further comprising:

instructions to provide a user interface comprising a data-source selection control, a query-criterion control, and a result-field control; and

instructions to receive the job specification via the user interface.