DATAFLOW OPTIMIZATION FOR EXTRACTIONS FROM A DATA REPOSITORY

• US 20160103898A1
• Filed: 06/24/2015
• Published: 04/14/2016
• Est. Priority Date: 10/10/2014
• Status: Active Grant

• Current Assignee: Intel Corporation, Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: ZHENG, ZUYE, MOKTALI, AMRUTA, DESHPANDE, SUYOG ANIL
• Granted Patent: US 10,380,136 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/289 Object oriented databases
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence

Abstract

Dataflow optimization is described for extractions from a data depository. In one example an object-relationship graph of a data extraction definition is traversed in a first pass. The object-relationship graph has a node for each object. The steps from each node of the graph in the first pass is determined. It is determined if any of the determined steps are repeated. A single instance of each repeated step is placed before other steps in a query plan. Data is then extracted in a second pass from the object-oriented database system according to the object-relationship graph by performing the repeated steps first and then reusing the repeated steps in other determined steps.

Claims

1. A database system-implemented method comprising:

traversing an object-relationship graph of a data extraction definition in a first pass, the object-relationship graph having a node for each object;

determining steps from each node of the graph in the first pass;

determining if any of the determined steps are repeated;

placing a single instance of each repeated step before other steps in a query plan;

extracting data in a second pass from an object-oriented database system according to the object-relationship graph by performing the repeated steps first and then reusing the repeated steps in other determined steps.

2. The method of claim 1, further comprising removing repeated instance of the repeated steps.

3. The method of claim 1, wherein the object-relationship graph defines relationships, joins and fetches of the data extraction definition.

4. The method of claim 3, further comprising selecting a super set of field objects of the object-relationship graph and using a subset of the selected super set for specific joins in the graph.

5. The method of claim 1, wherein the data extraction definition comprises sequential steps whereby the results of each step may be used by one or more other determined steps.

6. The method of claim 1, wherein the object-relationship graph defines a data extraction from an object-oriented database as steps through multiple objects as nodes of the graph and through relationships between the nodes through multiple passes.

8. The method of claim 1, further comprising performing further optimizations by deduping the object-relationship graph.

9. The method of claim 1, wherein the object relationship graph defines joins of the data extraction definition, the method further comprising selecting a super set of fields from the object-relationship graph and using a subset of the selected super set for specific joins in the graph.

7. A database system-implemented method comprising:

traversing an object-relationship graph through multiple passes, the object-relationship graph defining a data extraction from an object-oriented database as steps through multiple objects as nodes of the graph and through relationships between the nodes;

putting shared and reusable steps at the top of the data extraction definition; and

reusing shared and reusable steps in later steps of the data extraction definition.

10. A non-transitory machine-readable medium comprising a plurality of instructions which, when executed by a processing device, cause the processing device to perform one or more operations comprising:

traversing an object-relationship graph of a data extraction definition in a first pass, the object-relationship graph having a node for each object;

determining steps from each node of the graph in the first pass;

determining if any of the determined steps are repeated;

placing a single instance of each repeated step before other steps in a query plan;

extracting data in a second pass from an object-oriented database system according to the object-relationship graph by performing the repeated steps first and then reusing the repeated steps in other determined steps.

11. The medium of claim 10, the operations further comprising removing repeated instance of the repeated steps.

12. The medium of claim 10, wherein the object-relationship graph defines relationships, joins and fetches of the data extraction definition.

13. The medium of claim 12, the operations further comprising selecting a super set of field objects of the object-relationship graph and using a subset of the selected super set for specific joins in the graph.

14. The medium of claim 10, wherein the data extraction definition comprises sequential steps whereby the results of each step may be used by one or more other determined steps.

15. A system comprising:

a processor and a memory to execute instructions at the system; and

a mechanism to;

traverse an object-relationship graph through multiple passes, the object-relationship graph defining a data extraction from an object-oriented database as steps through multiple objects as nodes of the graph and through relationships between the nodes;

put shared and reusable steps at the top of the data extraction definition; and

reuse shared and reusable steps in later steps of the data extraction definition.

16. The system of claim 15, the mechanism further to perform further optimizations by deduping the object-relationship graph.

17. The system of claim 15, wherein the object relationship graph defines joins of the data extraction definition, the mechanism further to select a super set of fields from the object-relationship graph and using a subset of the selected super set for specific joins in the graph.