SYSTEM AND METHOD FOR OPERATING A BIG-DATA PLATFORM

• US 20130124483A1
• Filed: 11/08/2012
• Published: 05/16/2013
• Est. Priority Date: 11/10/2011
• Status: Abandoned Application

• Current Assignee: Treasure Data, Inc. (SoftBank Group Corp.)
• Original Assignee: Treasure Data, Inc. (SoftBank Group Corp.)
• Inventors: Furuhashi, Sadayuki, Yoshikawa, Hironobu, Ota, Kazuki
• Application Number:
  US13/671,896
• Publication Number:
  US 20130124483A1
• Time in Patent Office: Days
• US Class Current: 707/661
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A system and method for operating a big-data platform that includes at a data analysis platform, receiving discrete client data; storing the client data in a network accessible distributed storage system that includes: storing the client data in a real-time storage system; and merging the client data into a columnar-based distributed archive storage system; receiving a data query request through a query interface; and selectively interfacing with the client data from the real-time storage system and archive storage system according to the query.

Claims

1. A method for operating a big-data platform comprising:

at a data analysis platform, receiving discrete client data;

storing the client data in a network accessible distributed storage system that includes;

storing the client data in a real-time storage system;

merging the client data into a columnar-based distributed archive storage system;

receiving a data query request through a query interface; and

selectively interfacing with the client data from the real-time storage system and archive storage system according to the query.

2. The method of claim 1, wherein selectively interfacing with the client data includes at a data-intensive processing cluster, processing the data query according to a data mapping and reduction processes in querying data from the real-time storage system and archive storage system.

3. The method of claim 2, wherein the data-intensive processing cluster is a Hadoop cluster and the data mapping and reduction process is a MapReduce implementation.

4. The method of claim 1, wherein discrete client data is received and stored with dynamic schema.

5. The method of claim 4, wherein the data query request includes a schema definition and wherein selectively interfacing with the client data includes applying the schema definition to the dynamic schema.

6. The method of claim 1, further comprising at a client data agent collecting client data and transmitting the client data to the data analysis platform.

7. The method of claim 6, wherein the client data agent is integrated into an event channel from which client data is collected.

8. The method of claim 7, wherein the event channel is selected from the list consisting of syslog, a relational database, cloud data, and sensor data.

9. The method of claim 6, further comprising at the client data agent serializing data into a binary serialization data-interchange that is transmitted to the data analysis platform.

10. The method of claim 6, wherein collecting client data is collected through a client agent data-input plugin.

11. The method of claim 1, wherein the columnar-based distributed archive storage system stores client data in time series order, and wherein selectively interfacing with client data includes querying data from distributed storage system.

12. The method of claim 11, wherein querying client data from distributed storage system includes cooperatively querying the real-time storage system and the archive storage system for a cohesive query result.

13. The method of claim 1, wherein receiving a data query includes converting relational database styled query to data-intensive cluster query process.

14. The method of claim 1, wherein the data query request is received through a infographics interface and further comprising returning an infographic from the selectively interfaced client data.

15. The method of claim 1, wherein receiving a data query includes receiving the data query through a business intelligence tool driver and further comprising returning data analytics results to the business intelligence tool driver.

16. The method of claim 1, wherein client data is associated with a user account through unique identifier.

17. The method of claim 16, [scalable querying and isolated data storage], wherein client data merged into the archive data storage is isolated according to the user account associated with the client data and a query processing cluster interfaces with the distributed storage system, and the query processing cluster is shared between by a plurality of user accounts.

18. The method of claim 1, further comprising at a client data agent collecting client data and transmitting the client data to the data analysis platform;

wherein the columnar-based distributed archive storage system stores client data in time series order with a dynamic schema, and wherein selectively interfacing with client data includes cooperatively querying data from the real-time storage system and the archive storage system for a cohesive query result.

19. The method of claim 18, wherein distributed storage system includes over one petabyte of data.