Scalable analysis platform for semi-structured data

• US 10,275,475 B2
• Filed: 03/14/2014
• Issued: 04/30/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Tsirogiannis, Dimitris, Binkert, Nathan A., Harizopoulos, Stavros, Shah, Mehul A., Sowell, Benjamin A., Kaplan, Bryan D., Meyer, Kevin R.
• Primary Examiner(s): Raab, Christopher J
• Application Number:
  US14/210,495
• Publication Number:
  US 20140279834A1
• Time in Patent Office: 1,873 Days
• Field of Search: None
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/86 Mapping to a database

Abstract

A method of operating a data analysis system includes retrieving objects from a data source. Each of the retrieved objects includes (i) data and (ii) metadata describing the data. The method further includes dynamically creating a cumulative schema by, for each object of the retrieved objects: (i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object, (ii) creating a unified schema, wherein the unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema, and (iii) storing the unified schema as the cumulative schema. The method further includes exporting the data of each of the retrieved objects to a data warehouse.

Claims

1. A method of operating a data analysis system, the method comprising:

retrieving objects from a data source, wherein each of the retrieved objects includes (i) data and (ii) metadata describing the data;

dynamically updating a cumulative schema, wherein said dynamically updating comprises, for each object of the retrieved objects;

(i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object, wherein, for at least one object of the objects, a structure of an inferred schema is different from another structure of another inferred schema for another object of the objects,

(ii) creating a unified schema based at least on a portion of the inferred schema for the object, wherein the unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema, and

(iii) storing the unified schema as the cumulative schema;

converting the cumulative schema into a relational schema; and

exporting, according to the relational schema, the data of each of the retrieved objects to a data warehouse.

2. The method of claim 1, wherein the dynamically creating is performed during a first pass through the retrieved objects, and wherein the exporting is performed during a second pass through the retrieved objects.

3. The method of claim 1, further comprising:

storing the data of each of the retrieved objects into an index storage service, wherein the data of each of the retrieved objects is exported from the index storage service to the data warehouse.

4. The method of claim 3, wherein the exporting includes:

creating at least one intermediate file from the index storage service, wherein the at least one intermediate file has a predefined data warehouse format; and

bulk loading the at least one intermediate file into the data warehouse.

5. The method of claim 4, wherein the at least one intermediate file is created according to the relational schema.

6. The method of claim 3, further comprising:

receiving a query from a user via a graphical user interface; and

responding to the query based on at least one of (i) data stored by the index storage service and (ii) results returned from the data warehouse.

7. The method of claim 6, further comprising passing the query to the data warehouse in order to obtain the results.

8. The method of claim 6, further comprising:

displaying initial results to the user via the graphical user interface; and

iteratively updating results in the graphical user interface as execution of the query continues.

9. The method of claim 1, further comprising:

receiving a query from a user via a graphical user interface; and

responding to the query based on results returned from the data warehouse.

10. The method of claim 1, further comprising:

receiving a query from a user via a graphical user interface;

displaying initial results of the query to the user in the graphical user interface; and

iteratively updating results of the query in the graphical user interface as execution of the query continues.

11. The method of claim 10, wherein the updating results in the graphical user interface includes updating scaling of at least one axis of at least one data chart.

12. The method of claim 1, further comprising:

displaying the cumulative schema to a user via a graphical user interface;

updating the cumulative schema as additional data is retrieved from the data source; and

selectively updating the graphical user interface to reflect the updated cumulative schema.

13. The method of claim 12 further comprising, in the user interface, visually distinguishing changed items in the updated cumulative schema.

14. The method of claim 1, further comprising repeating the retrieving, the dynamically creating, and the exporting in response to new objects being available from the data source.

15. The method of claim 14, further comprising, prior to repeating the exporting:

determining whether the cumulative schema has changed since a previous exporting; and

in response to determining that the cumulative schema has changed, sending at least one command to the data warehouse to update a schema of the data warehouse to reflect the changes to the cumulative schema.

16. A non-transitory computer-readable medium storing processor-executable instructions comprising:

retrieving objects from a data source, wherein each of the retrieved objects includes (i) data and (ii) metadata describing the data;

dynamically updating a cumulative schema, wherein said dynamically updating comprises, for each object of the retrieved objects;

(i) inferring a schema from the object based on the metadata of the object and inferred data types of elements of the data of the object, wherein, for at least one object of the objects, a structure of an inferred schema is different from another structure of another inferred schema for another object of the objects,

(ii) creating a unified schema based at least on a portion of the inferred schema for the object, wherein the unified schema describes both (a) the object described by the inferred schema and (b) a cumulative set of objects described by the cumulative schema, and

(iii) storing the unified schema as the cumulative schema;

converting the cumulative schema into a relational schema; and

exporting, according to the relational schema, the data of each of the retrieved objects to a data warehouse.

17. The computer-readable medium of claim 16, wherein the instructions further comprise storing the data of each of the retrieved objects into an index storage service, wherein the data of each of the retrieved objects is exported from the index storage service to the data warehouse.

18. The computer-readable medium of claim 16, wherein the instructions further comprise:

displaying the cumulative schema to a user via a graphical user interface;

updating the cumulative schema as additional data is retrieved from the data source; and

selectively updating the graphical user interface to reflect the updated cumulative schema.