Methods for accessing big data and systems using the same

• US 9,600,491 B2
• Filed: 10/29/2014
• Issued: 03/21/2017
• Est. Priority Date: 08/01/2014
• Status: Active Grant

• Current Assignee: Wistron Corporation
• Original Assignee: Wistron Corporation
• Inventors: Chen, Chih-Ming
• Primary Examiner(s): Merant, Guerrier
• Application Number:
  US14/527,694
• Publication Number:
  US 20160034491A1
• Time in Patent Office: 874 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems

Abstract

An embodiment of the invention introduces a method for accessing big data, which contains at least the following steps. A data access request is received from one of a plurality of database frontends of different kinds. A data access operation is performed for the data access request by using an API (Application Programming Interface) to manipulate one of a plurality of cloud file systems of different kinds.

Claims

1. A method for accessing big data, comprising:

dispatching a DB (Database) API (Application Programming Interface) request from a client to one of plurality of proxy modules, wherein the proxy module receiving the DB API request connects to a database frontend, wherein the database frontend is one of a plurality of database frontends of different kinds, wherein the database frontend interprets the DB API request from the client and issues a data access request;

receiving the data access request from the database frontend; and

performing a data access operation for the data access request by using an API to manipulate one of a plurality of cloud file systems of different kinds, thereby enabling any of the database frontends of different kinds to use unified access methods to request information of the cloud file systems of different kinds.

2. The method of claim 1, wherein the API is associated with a cross-platform document-oriented database and the manipulated cloud file system is a grid file system.

3. The method of claim 1, wherein the API is associated with a distributed batch database and the manipulated cloud file system is a distributed file system.

4. The method of claim 1, wherein the API is associated with a distributed real-time database and the manipulated cloud file system is a distributed file system.

5. The method of claim 1, wherein the data access request is a data write request, the method further comprising:

generating an ECC (Error Check and Correction) code according to data to be written, wherein the data to be written is stored in a data fragment of a file and the ECC code is stored in a parity fragment of the file; and

writing values of the data fragment and the parity fragment to the cloud file system by using the API.

6. The method of claim 5, further comprising:

computing a hash value for each of the data fragment and the parity fragment; and

storing the hash values to the data fragment and the parity fragment.

7. The method of claim 6, further comprising:

storing a meta tag to the data fragment for recognizing a data type of the data fragment.

8. The method of claim 7, wherein the data access request is a data read request, the method further comprising:

reading data from the cloud file system using the API, wherein the data comprises values of a data fragment and a parity fragment;

determining whether the read data is correct;

replying with the read data when the read data is correct;

performing an error correction algorithm to attempt to correct error bits of the read data and determining whether the error bits have been corrected successfully when the read data is not correct;

replying with the corrected data when the error bits have been corrected successfully; and

replying with a data read failure message when the error bits have not been corrected successfully.

9. The method of claim 8, further comprising:

writing the corrected data back to the cloud file system by using the API when the error bits have been corrected successfully.

10. The method of claim 9, further comprising:

sending the meta tag of the corrected data fragment to a SIA (Security Intelligence Analytics) module when the error bits have been corrected successfully, wherein the sent meta tag is used to recognize a data type of the corrected data fragment.

11. The method of claim 10, further comprising:

receiving a specified number of the same meta tag;

publishing a fake document as a honeypot by the SIA module.

12. The method of claim 10, wherein the SIA module predicts what categories of data are hotspots for hackers.

13. A system for accessing big data, comprising:

a virtual server dispatching a DB (Database) API (Application Programming Interface) request from a client to one of plurality of proxy modules, wherein the proxy module receiving the DB API request connects to a database frontend, wherein the database frontend is one of a plurality of database frontends of different kinds, wherein the database frontend interprets the DB API request from the client and issues a data access request; and

a SVH (Secure Verification Hashing) module, coupled between the database frontend and a plurality of cloud file systems of different kinds, receiving the data access request from the database frontend; and

performing a data access operation for the data access request by using an API to manipulate one of the cloud file systems, thereby enabling any of the database frontends of different kinds to use unified access methods to request information of the cloud file systems of different kinds.

14. The system of claim 13, wherein the API is provided by one of a cross-platform document-oriented database, a distributed batch database and a distributed real-time database, and the cloud file system comprises a grid file system and a distributed file system.

15. The system of claim 13, wherein the data access request is a data write request, the system further comprising:

an erasure coding module, generating an ECC (Error Check and Correction) code according to data to be written, wherein the data to be written is stored in a data fragment of a file and the ECC code is stored in a parity fragment of the file; and

writing values of the data fragment and the parity fragment to the cloud file system by using the API.

16. The system of claim 15, wherein the SVH module computes a hash value for each of the data fragment and the parity fragment and stores the hash values to the data fragment and the parity fragment.

17. The system of claim 16, wherein the SVH module stores a meta tag for the data fragment for recognizing a data type of the data fragment.

18. The system of claim 13, wherein the data access request is a data read request and the SVH module reads data from the cloud file system using the API, where the data comprises values of a data fragment and a parity fragment;

determines whether the read data is correct;

replies with the read data when the read data is correct.

19. The system of claim 18, wherein the SVH module performs an error correction algorithm to attempt to correct error bits of the read data and determines whether the error bits have been corrected successfully when the read data is not correct;

replies with the corrected data when the error bits have been corrected successfully; and

replies with a data read failure message when the error bits have not been corrected successfully.

20. The system of claim 19, wherein the SVH module writes the corrected data back to the cloud file system using the API when the error bits have been corrected successfully.

21. The system of claim 20, wherein the SVH module sends the meta tag of the corrected data fragment to a SIA (Security Intelligence Analytics) module when the error bits have been corrected successfully, and the sent meta tag is used to recognize a data type of the corrected data fragment.

22. The system of claim 21, wherein, after receiving a specified number of the same meta tag, the SIA module publishes a fake document as a honeypot.

23. The system of claim 21, wherein the SIA module predicts what categories of data are hotspots for hackers.