Method, system, and apparatus for enterprise wide storage and retrieval of large amounts of data

• US 9,747,350 B2
• Filed: 07/23/2014
• Issued: 08/29/2017
• Est. Priority Date: 12/08/2010
• Status: Active Grant

• Current Assignee: YottaStor, LLC
• Original Assignee: YottaStor, LLC
• Inventors: Carlson, Robert John
• Primary Examiner(s): Ibrahim, Mohamed
• Application Number:
  US14/338,785
• Publication Number:
  US 20140337336A1
• Time in Patent Office: 1,133 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: H04L 67/1042 using topology management mechanisms
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A scalable network of mobile data storage containers that are connected in peer-to-peer networks to archive large data storage capacities. The various embodiments provide a method of extracting a large amount of data from a variety of sources and storing the extracted data in mobile, storage units. The various embodiments provide storage units housed in mobile containers that can store multiple days/weeks of sensor data in the order of petabytes (1024 terabytes). The various embodiments, integrate high performance computing devices into the mobile storage containers that are able to perform critical extraction, pattern, and index processing on the sensor data. The various embodiments, provide a method for the efficient physical transport of the mobile storage containers from current locations to a center analysis location for re-connecting in another peer-to-peer network for integration into a central enterprise data warehouses.

Claims

1. A method for collecting, storing, and analyzing information, comprising:

provisioning a first mobile data storage unit at a first location, the provisioning comprising installation of an analysis software to ingest and process a single data type or a related family of data types in a namespace;

transporting the first mobile data storage unit to an area of operations at a second location;

connecting the first mobile data storage unit with one or more mobile data storage units in a peer-to-peer network;

collecting information in the area of operations;

wherein the first mobile data storage unit is optimized for a single data type or related family of data types to ingest the data types, search the meta data for the data types and disseminate the data types to other processing and application system environments; and

wherein the first mobile storage unit is assigned a namespace ID, wherein data objects ingested into the namespace ID are assigned unique identifiers, wherein the namespace ID is persistent through disconnection and reconnection of the first mobile storage unit, and wherein the peer-to-peer network is configured to assign data processing capabilities to the mobile storage unit based on the namespace ID.

2. The method of claim 1, further comprising:

pairing the first mobile storage unit with one or more mobile data storage units in the peer-to-peer network;

replicating the data of the first mobile storage unit on the one or more paired mobile storage units; and

removing the first mobile storage unit from the peer-to-peer network;

wherein the one or more paired mobile storage units remain in operation on the peer-to-peer network.

3. The method of claim 2, further comprising:

connecting a replacement mobile data storage unit with one or more mobile data storage units in the peer-to-peer network; and

automatically replicating the data of the first mobile storage unit on the replacement mobile data storage unit.

4. The method of claim 1, wherein a pool of servers across multiple mobile data storage units are dynamically allocated to the analysis software processing internal data.

5. The method of claim 1, wherein at least one second analysis software can be provisioned in the first mobile data storage unit.

6. The method of claim 1, wherein the data stored in one mobile data storage unit can be accessed by analysis software from another mobile data storage unit.

7. The method of claim 1, wherein analysis software extracts the meta data from the single data type or related family of data types and creates an index of this meta data that can be searched.

8. The method of claim 1, wherein the optimized software writes the data as an object in a unique namespace for all mobile data storage units in the peer-to-peer network.

9. A method for embedding analytical software in a mobile data storage unit in a peer-to-peer network, comprising:

provisioning a first mobile data storage unit at a first location, the provisioning comprising installation of an analysis software;

transporting the first mobile data storage unit to an area of operations at a second location;

connecting a plurality of second mobile data storage units with the first data storage unit in a first peer-to-peer network;

processing data in each of the mobile data storage units independently of the processing of data in the other mobile data storage units in the peer-to-peer network;

aggregating the results from the independent processing performed in each of the mobile data storage units from all the mobile data storage units in the peer-to-peer network into a preliminary data set; and

transmitting the preliminary data set to one or more analytical environments for additional processing;

wherein the one or more analytical environments can perform different processing actions on the preliminary data set using different analytical techniques and approaches as required by a business process; and

wherein the first mobile storage unit is assigned a namespace ID, wherein data objects ingested into the namespace ID are assigned unique identifiers, wherein the namespace ID is persistent through disconnection and reconnection of the first mobile storage unit, and wherein the peer-to-peer network is configured to assign data processing capabilities to the mobile storage unit based on the namespace ID.

10. The method of claim 9, further comprising:

pairing the first mobile storage unit with one or more mobile data storage units in the peer-to-peer network;

replicating the data of the first mobile storage unit on the one or more paired mobile storage units; and

removing the first mobile storage unit from the peer-to-peer network;

wherein the one or more paired mobile storage units remain in operation on the peer-to-peer network.

11. The method of claim 10, further comprising:

connecting a replacement mobile data storage unit with one or more mobile data storage units in the peer-to-peer network; and

automatically replicating the data of the first mobile storage unit on the replacement mobile data storage unit.

12. The method of claim 9, wherein the one or more analytical environments can be another processing node on the peer-to-peer network or be a separate environment on a separate network.

13. A method for embedding analytical software and applications in mobile data storage unit in a peer-to-peer network for the purpose of processing the data without moving the data from the mobile data storage unit within the peer-to-peer network, comprising:

provisioning a first mobile data storage unit at a first location, the provisioning comprising installation of an analysis software;

connecting a plurality of mobile data storage units with the first data storage unit in a first peer-to-peer network

transporting the first mobile data storage unit to an area of operations at a second location;

processing data in each of the mobile data storage units independently of the other mobile data storage units in the peer-to-peer network, wherein the analytical software in each of the mobile data center storage units is able to access and utilize compute resources in other mobile data storage units; and

aggregating analytical results from each mobile data storage unit and delivering the analytical results to an application for presentation to a user;

wherein the first mobile storage unit is assigned a namespace ID, wherein data objects ingested into the namespace ID are assigned unique identifiers, wherein the namespace ID is persistent through disconnection and reconnection of the first mobile storage unit, and wherein the peer-to-peer network is configured to assign data processing capabilities to the mobile storage unit based on the namespace ID.

14. The method of claim 13, further comprising:

pairing the first mobile storage unit with one or more mobile data storage units in the peer-to-peer network;

replicating the data of the first mobile storage unit on the one or more paired mobile storage units; and

removing the first mobile storage unit from the peer-to-peer network;

wherein the one or more paired mobile storage units remain in operation on the peer-to-peer network.

15. The method of claim 14, further comprising:

connecting a replacement mobile data storage unit with one or more mobile data storage units in the peer-to-peer network; and

automatically replicating the data of the first mobile storage unit on the replacement mobile data storage unit.

16. The method of claim 13, wherein an activation of the analysis software can be achieved by any user who has access to the peer-to-peer network.

17. The method of claim 13, wherein at least one processor operable to execute the analysis software within the mobile storage units is dynamically allocated to the analysis software for processing data.