MOBILE HADOOP CLUSTERS

• US 20140040575A1
• Filed: 08/01/2012
• Published: 02/06/2014
• Est. Priority Date: 08/01/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Horn, Gustav
• Granted Patent: US 9,223,845 B2
• Time in Patent Office: Days
• US Class Current: 711/162
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 16/28 Databases characterised by their database models, e.g. relational or object models
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays

Abstract

Techniques for mobile clusters for collecting telemetry data and processing analytic tasks, are disclosed herein. The mobile cluster includes a processor, a plurality of data nodes and an analysis module. The data nodes receive and store a snapshot of at least a portion of data stored in a main Hadoop storage cluster and real-time acquired data received from a data capturing device. The analysis module is operatively coupled to the processor to process analytic tasks based on the snapshot and the real-time acquired data when the storage cluster is not connected to the main storage cluster.

Claims

1. A storage cluster comprising:

a processor;

a plurality of data nodes for receiving and storing a snapshot of at least a portion of data stored in a main storage cluster and acquired data received from a data capturing device; and

an analysis module operatively coupled to the processor to process analytic tasks based on the snapshot and the acquired data when the storage cluster is not connected to the main storage cluster.

2. The storage cluster of claim 1, further comprising:

a connection; and

an integration module to integrate the acquired data into the main storage cluster via the connection.

3. The storage cluster of claim 1, wherein the storage cluster is disposed in a vehicle, and the data capturing device is disposed in or on the vehicle.

4. The storage cluster of claim 3, wherein the analysis module is configured to process analytic tasks from a client device within the vehicle based on the snapshot and the acquired data without being connected to the main storage cluster.

5. The storage cluster of claim 1, further comprising:

a connection to retrieve incoming data from the main storage cluster; and

a snapshot module to generate the snapshot of at least the portion of the data store in the main storage cluster based on the incoming data.

6. The storage cluster of claim 1, wherein the acquired data includes telemetry data, and the data capturing device comprises a telemetry data collecting device.

7. The storage cluster of claim 1, wherein the data nodes are removable from the storage cluster and configured to be integrated into the main storage cluster.

8. The storage cluster of claim 1, wherein the storage cluster is a Hadoop storage cluster, which is separated from the main storage cluster when the storage cluster receives the acquired data.

9. The storage cluster of claim 5, wherein, when the storage cluster is connected to the main storage cluster via the connection, the snapshot module is configured to further receive an update via the connection from the main storage cluster and to apply the update to the snapshot;

and wherein the update includes changes of the data stored in the main storage cluster since a last time the storage cluster was connected to the main storage cluster.

10. The storage cluster of claim 1, wherein the main storage cluster is a Hadoop storage cluster.

11. A Hadoop cluster comprising:

a processor;

a plurality of data nodes for storing data;

a snapshot module operatively coupled to the processor to generate a snapshot of at least a portion of the data stored in the data nodes; and

a connection configured to transfer the snapshot to a Mobile cluster.

12. The Hadoop cluster of claim 11, further comprising:

an integration module stored in a memory of the Hadoop cluster that is executable by the processor to instantly integrate real-time acquired data from the mobile cluster into the Hadoop cluster via the connection, when the mobile cluster is connected to the Hadoop cluster via the connection.

13. The Hadoop cluster of claim 12, wherein the real-time acquired data is telemetry data collected into the mobile cluster by a telemetry data collecting device when the mobile cluster is not connected to the Hadoop cluster.

14. The Hadoop cluster of claim 11, wherein the mobile cluster is disposed in a vehicle when the mobile cluster is not connected to the Hadoop cluster, and the real-time acquired data is received from a data capturing device within the vehicle.

15. The Hadoop cluster of claim 11, wherein the snapshot enables the mobile cluster to process analytic tasks based on the snapshot without a need to connect to the Hadoop cluster.

16. The Hadoop cluster of claim 11, wherein the Hadoop cluster is configured to integrate a plurality of mobile data nodes removable from the mobile cluster.

17. The Hadoop cluster of claim 11, wherein, when the mobile cluster is connected to the main Hadoop cluster via the connection, the snapshot module is executable by the processor to further generate an update of the data and transmit the update via the connection to the mobile cluster;

and wherein the update includes changes of the data stored in the Hadoop cluster since last time the mobile cluster was connected to the Hadoop cluster.

18. A computer implemented method comprising:

receiving, at a mobile storage cluster, a snapshot of at east a portion of data stored in a main storage cluster;

collecting, at the mobile storage cluster, real-time acquired data from a data capturing device; and

processing analytic tasks in the mobile storage cluster based on the snapshot and the real-time acquired data when the mobile storage cluster is not connected to the main storage cluster.

19. The method of claim 18, further comprising:

integrating the real-time acquired data into the main storage cluster instantly via a connection when the mobile storage cluster is connected to the main storage cluster.

20. The method of claim 18, wherein said collecting real-time acquired data comprises:

collecting real-time acquired data from a data capturing device disposed in or on a vehicle.

21. The method of claim 18, wherein said processing analytic tasks comprises:

processing analytic tasks from a client device in the mobile storage cluster based on the snapshot and the real-time acquired data when the mobile storage cluster is not connected to the main storage cluster, when the device and the mobile storage cluster are disposed in a vehicle.

22. The method of claim 18, further comprising:

generating the snapshot of at least the portion of data stored in the main storage cluster.

23. The method of claim 18, wherein the real-time acquired data is telemetry data, and the data capturing device is a telemetry data collecting device.

24. The method of claim 18, further comprising:

removing data nodes storing the real-time acquired data from the mobile storage cluster; and

integrating the data nodes into the main storage cluster.

25. The method of claim 18, further comprising:

receiving an update from the main storage cluster when the mobile storage cluster is connected to the main storage cluster, wherein the update includes changes of the data stored in the main storage cluster since a last time when the mobile storage cluster was connected to the main storage cluster; and

applying the update to the snapshot stored in the mobile storage cluster.

26. The method of claim 18, wherein the main storage cluster is a Hadoop storage cluster.

27. A data storage node comprising:

a processor;

a connection interface through which to contact one or more other data storage nodes within a mobile storage cluster for accessing a snapshot of at least a portion of data stored in a main storage cluster; and

an analysis module operatively coupled to the processor to process analytic tasks based on the snapshot and acquired data received from a data capturing device, when the mobile storage cluster is not connected to the main storage cluster.

28. The data storage node of claim 27, further comprising:

are integration module to integrate the acquired data into the main storage cluster when the mobile storage cluster is connected to the main storage cluster.

29. The data storage node of claim 27, wherein the analysis module is configured to process analytic tasks from a client device within a vehicle based on the snapshot and the acquired data when the mobile storage cluster is not connected to the main storage cluster.

30. The data storage node of claim 27, wherein the mobile storage cluster is a mobile Hadoop storage cluster, which is separated from the main storage cluster when the mobile Hadoop storage cluster receives the acquired data.

31. A computer Implemented method comprising:

receiving, at a data storage node of a mobile storage cluster, an analytic task from a client device;

contacting one or more other data storage nodes within a main storage cluster for accessing a snapshot of at least a portion of data stored in the main storage cluster; and

processing the analytic task, at the mobile storage cluster, based on the snapshot and acquired data from a data capturing device, when the mobile storage cluster is not connected to the main storage cluster.