Mobile hadoop clusters

• US 9,223,845 B2
• Filed: 08/01/2012
• Issued: 12/29/2015
• Est. Priority Date: 08/01/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Horn, Gustav
• Primary Examiner(s): Rojas, Midys
• Application Number:
  US13/564,622
• Publication Number:
  US 20140040575A1
• Time in Patent Office: 1,245 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 16/28 Databases characterised by their database models, e.g. relational or object models
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays

Abstract

Techniques for mobile clusters for collecting telemetry data and processing analytic tasks, are disclosed herein. The mobile cluster includes a processor, a plurality of data nodes and an analysis module. The data nodes receive and store a snapshot of at least a portion of data stored in a main Hadoop storage cluster and real-time acquired data received from a data capturing device. The analysis module is operatively coupled to the processor to process analytic tasks based on the snapshot and the real-time acquired data when the storage cluster is not connected to the main storage cluster.

Claims

1. A storage cluster, comprising:

a processor;

a plurality of data nodes for receiving and storing a point-in-time image of at least a portion of data stored in a main storage cluster, that is separate from the storage cluster, and acquired data received from a data capturing device; and

an analysis module operatively coupled to the processor to process analytic tasks based on the point-in-time image and the acquired data when the storage cluster is not connected to the main storage cluster for exchanging data;

wherein the data nodes of the storage cluster are removable from the storage cluster and configured to be integrated into the main storage cluster, and in an event that the data nodes are integrated into the main storage cluster, the main storage cluster uses differences of file system metadata between the storage cluster and the main storage cluster to update a file system of the main storage cluster.

2. The storage cluster of claim 1, further comprising:

a connection; and

an integration module to integrate the acquired data into the main storage cluster via the connection.

3. The storage cluster of claim 1, wherein the storage cluster is disposed in a vehicle, and the data capturing device is disposed in or on the vehicle.

4. The storage cluster of claim 3, wherein the analysis module is configured to process analytic tasks from a client device within the vehicle based on the point-in-time image and the acquired data without being connected to the main storage cluster.

5. The storage cluster of claim 1, further comprising:

a connection to retrieve incoming data from the main storage cluster; and

a point-in-time image module to generate the point-in-time image of at least the portion of the data store in the main storage cluster based on the incoming data.

6. The storage cluster of claim 5, wherein, when the storage cluster is connected to the main storage cluster via the connection, the point-in-time image module is configured to further receive an update via the connection from the main storage cluster and to apply the update to the point-in-time image;

and wherein the update includes changes of the data stored in the main storage cluster since a last time the storage cluster was connected to the main storage cluster.

7. The storage cluster of claim 1, wherein the acquired data includes telemetry data, and the data capturing device comprises a telemetry data collecting device.

8. The storage cluster of claim 1, wherein the storage cluster is a Hadoop storage cluster, which is separated from the main storage cluster when the storage cluster receives the acquired data.

9. The storage cluster of claim 1, wherein the main storage cluster is a Hadoop storage cluster.

10. A computational cluster for analyzing unstructured data, comprising:

a processor;

a plurality of data nodes for storing unstructured data;

a point-in-time image module operatively coupled to the processor to generate a point-in-time image of at least a portion of the unstructured data stored in the data nodes;

a connection configured to transfer the point-in-time image to a mobile cluster; and

an integration module stored in a memory of the computational cluster that is executable by the processor to instantly integrate real-time acquired data from the mobile cluster into the computational cluster via the connection, when the mobile cluster is connected to the computational cluster via the connection;

wherein integration module is further configured to use differences of file system metadata between the computational cluster and the mobile cluster to update a file system of the computational cluster.

11. The computational cluster of claim 10, wherein the real-time acquired data is telemetry data collected into the mobile cluster by a telemetry data collecting device when the mobile cluster is not connected to the computational cluster.

12. The computational cluster of claim 10, wherein the mobile cluster is disposed in a vehicle when the mobile cluster is not connected to the computational cluster, and the real-time acquired data is received from a data capturing device within the vehicle.

13. The computational cluster of claim 10, wherein the point-in-time image enables the mobile cluster to process analytic tasks based on the point-in-time image without a need to connect to the computational cluster.

14. The computational cluster of claim 10, wherein the computational cluster is configured to integrate a plurality of mobile data nodes removable from the mobile cluster.

15. The computational cluster of claim 10, wherein, when the mobile cluster is connected to the main computational cluster via the connection, the point-in-time image module is executable by the processor to further generate an update of the unstructured data and transmit the update via the connection to the mobile cluster;

and wherein the update includes changes of the data stored in the computational cluster since last time the mobile cluster was connected to the computational cluster.

16. The computational cluster of claim 10, wherein the computational cluster is a Hadoop cluster.

17. A computer implemented method, comprising:

receiving, at a mobile storage cluster, a point-in-time image of at least a portion of data stored in a main storage cluster;

collecting, at the mobile storage cluster, real-time acquired data from a data capturing device;

processing analytic tasks in the mobile storage cluster based on the point-in-time image and the real-time acquired data when the mobile storage cluster is not connected to the main storage cluster;

removing data nodes storing the real-time acquired data from the mobile storage cluster;

integrating the data nodes into the main storage cluster; and

using differences of file system metadata between the mobile storage cluster and the main storage cluster to update a file system of the main storage cluster.

18. The method of claim 17, further comprising:

integrating the real-time acquired data into the main storage cluster instantly via a connection when the mobile storage cluster is connected to the main storage cluster.

19. The method of claim 17, wherein said collecting real-time acquired data comprises:

collecting real-time acquired data from a data capturing device disposed in or on a vehicle.

20. The method of claim 17, wherein said processing analytic tasks comprises:

processing analytic tasks from a client device in the mobile storage cluster based on the point-in-time image and the real-time acquired data when the mobile storage cluster is not connected to the main storage cluster, when the device and the mobile storage cluster are disposed in a vehicle.

21. The method of claim 17, further comprising:

generating the point-in-time image of at least the portion of data stored in the main storage cluster.

22. The method of claim 17, wherein the real-time acquired data is telemetry data, and the data capturing device is a telemetry data collecting device.

23. The method of claim 17, further comprising:

receiving an update from the main storage cluster when the mobile storage cluster is connected to the main storage cluster, wherein the update includes changes of the data stored in the main storage cluster since a last time when the mobile storage cluster was connected to the main storage cluster; and

applying the update to the point-in-time image stored in the mobile storage cluster.

24. The method of claim 17, wherein the main storage cluster is a Hadoop storage cluster.

25. A data storage node, comprising:

a processor;

a connection interface through which to contact one or more other data storage nodes within a mobile storage cluster for accessing a point-in-time image of at least a portion of data stored in a main storage cluster;

an analysis module operatively coupled to the processor to process analytic tasks based on the point-in-time image and acquired data received from a data capturing device, when the mobile storage cluster is not connected to the main storage cluster; and

an integration module to integrate the acquired data into the main storage cluster when the mobile storage cluster is connected to the main storage cluster;

wherein integration module is further configured to use differences of file system metadata between the main storage cluster and the mobile storage cluster to update a file system of the main storage cluster.

26. The data storage node of claim 25, wherein the analysis module is configured to process analytic tasks from a client device within a vehicle based on the point-in-time image and the acquired data when the mobile storage cluster is not connected to the main storage cluster.

27. The data storage node of claim 25, wherein the mobile storage cluster is a mobile Hadoop storage cluster, which is separated from the main storage cluster when the mobile Hadoop storage cluster receives the acquired data.

28. A computer implemented method, comprising:

receiving, at a data storage node of a mobile storage cluster, an analytic task from a client device;

contacting one or more other data storage nodes within a main storage cluster for accessing a point-in-time image of at least a portion of data stored in the main storage cluster; and

processing the analytic task, at the mobile storage cluster, based on the point-in-time image and acquired data from a data capturing device, when the mobile storage cluster is not connected to the main storage cluster;

removing data nodes storing the real-time acquired data from the mobile storage cluster;

integrating the data nodes into the main storage cluster; and

using differences of file system metadata between the mobile storage cluster and the main storage cluster to update a file system of the main storage cluster.