SYSTEM AND METHOD FOR OPERATING A BIG-DATA PLATFORM

• US 20160246824A1
• Filed: 05/05/2016
• Published: 08/25/2016
• Est. Priority Date: 11/10/2011
• Status: Active Grant

• Current Assignee: Treasure Data, Inc. (SoftBank Group Corp.)
• Original Assignee: Treasure Data, Inc. (SoftBank Group Corp.)
• Inventors: Yoshikawa, Hironobu, Furuhashi, Sadayuki, Ota, Kazuki
• Granted Patent: US 9,582,528 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A system and method for operating a big-data platform that includes at a data analysis platform, receiving discrete client data; storing the client data in a network accessible distributed storage system that includes: storing the client data in a real-time storage system; and merging the client data into a columnar-based distributed archive storage system; receiving a data query request through a query interface; and selectively interfacing with the client data from the real-time storage system and archive storage system according to the query.

Claims

1. A method for operating a big-data platform comprising:

at a data analysis platform, receiving discrete client data;

storing the client data in a network accessible distributed storage system that includes;

storing the client data in a real-time storage system in a row format;

merging the client data into a columnar-based distributed archive storage system;

receiving a data query through a query interface; and

processing the data query by selectively interfacing with the client data from the real-time storage system and archive storage system, according to a data mapping and reduction process, wherein processing the data query comprises cooperatively querying the real-time storage system and the archive storage system and distributing the data query over the real-time storage system and the archive storage system to retrieve a single cohesive query result,

wherein merging the client data into a columnar-based distributed archive storage system comprises storing the client data in the archive storage system in a columnar format, and

wherein interfacing with the client data from the archive storage system comprises;

converting, by using a query processing cluster, at least a portion of the data query to the mapping process and the reduction process; and

executing the mapping process and the reduction process by using the query processing cluster.

2. The method of claim 1, wherein discrete client data is received and stored with dynamic schema.

3. The method of claim 2, wherein the data query includes a schema definition and wherein selectively interfacing with the client data includes applying the schema definition to the dynamic schema.

4. The method of claim 1, further comprising at a client data agent collecting client data and transmitting the client data to the data analysis platform.

5. The method of claim 4, wherein the client data agent is integrated into an event channel from which client data is collected.

6. The method of claim 5, wherein the event channel is selected from the list consisting of syslog, a relational database, cloud data, and sensor data.

7. The method of claim 4, further comprising at the client data agent serializing data into a binary serialization data-interchange that is transmitted to the data analysis platform.

8. The method of claim 4, wherein collecting client data is collected through a client agent data-input plugin.

9. The method of claim 1, wherein the columnar-based distributed archive storage system stores client data in time series order, and wherein selectively interfacing with client data includes querying data from distributed storage system.

10. The method of claim 1, wherein receiving a data query includes converting relational a database styled query to a data-intensive cluster query process.

11. The method of claim 1, wherein the data query is received through an infographics interface and further comprising returning an infographic from the selectively interfaced client data.

12. The method of claim 1, wherein receiving a data query includes receiving the data query through a business intelligence tool driver and further comprising returning data analytics results to the business intelligence tool driver.

13. The method of claim 1, wherein client data is associated with a user account through unique identifier.

14. The method of claim 13, wherein client data merged into the archive data storage system is isolated according to the user account associated with the client data and the query processing cluster interfaces with the distributed storage system, and the query processing cluster is shared between by a plurality of user accounts.

15. The method of claim 1, further comprising at a client data agent collecting client data and transmitting the client data to the data analysis platform;

wherein the columnar-based distributed archive storage system stores client data in time series order with a dynamic schema, and wherein selectively interfacing with client data includes cooperatively querying data from the real-time storage system and the archive storage system for a cohesive query result.

16. The method of claim 15, wherein distributed storage system includes over one petabyte of data.

17. The method of claim 1, wherein the mapping process and the reduction process are MapReduce processes.

18. The method of claim 1, wherein the query processing cluster is constructed to execute MapReduce processes, and the mapping process and the reduction process are MapReduce processes.

19. The method of claim 1, wherein the query processing cluster includes a Hadoop enabled cluster that is constructed to execute MapReduce processes, and the mapping process and the reduction process are MapReduce processes.

20. The method of claim 1, wherein the data analysis platform is a multi-tenant data analysis platform.

21. The method of claim 1, wherein the query result includes structured data.

22. The method of claim 2, wherein the data analysis platform is a multi-tenant data analysis platform.

23. The method of claim 3, wherein the data analysis platform is a multi-tenant data analysis platform.

24. A method comprising:

at a multi-tenant data analysis platform;

receiving discrete client data, the client data being associated with a user account of the multi-tenant data analysis platform through a unique identifier;

storing the client data in a network accessible distributed storage system that includes a real-time storage system and a columnar-based distributed archive storage system, the storing of the client data comprising;

storing the client data in the real-time storage system in a row format;

merging the client data into the archive storage system in a columnar format, the client data merged into the archive data storage system being isolated according to the user account associated with the client data;

receiving a data query through a query interface; and

processing the data query by selectively interfacing with the client data from the real-time storage system and archive storage system, wherein processing the data query comprises cooperatively querying the real-time storage system and the archive storage system and distributing the data query over the real-time storage system and the archive storage system to retrieve a single cohesive query result,

wherein interfacing with the client data from the archive storage system comprises;

converting, by using a query processing cluster, the data query to a MapReduce mapping process and a MapReduce reduction process; and

executing the MapReduce mapping process and the MapReduce reduction process by using the query processing cluster, and

wherein the query processing duster includes a Hadoop enabled duster that is constructed to execute MapReduce processes.