Method, system, and apparatus for enterprise wide storage and retrieval of large amounts of data

• US 10,528,579 B2
• Filed: 07/28/2017
• Issued: 01/07/2020
• Est. Priority Date: 12/08/2010
• Status: Active Grant

• Current Assignee: YottaStor, LLC
• Original Assignee: YottaStor, LLC
• Inventors: Carlson, Robert John
• Primary Examiner(s): Ibrahim, Mohamed
• Application Number:
  US15/662,678
• Publication Number:
  US 20170344613A1
• Time in Patent Office: 893 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: H04L 67/1042 using topology management mechanisms
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A scalable network of mobile data storage containers that are connected in peer-to-peer networks to archive large data storage capacities. The various embodiments provide a method of extracting a large amount of data from a variety of sources and storing the extracted data in mobile, storage units. The various embodiments provide storage units housed in mobile containers that can store multiple days/weeks of sensor data in the order of petabytes (1024 terabytes). The various embodiments, integrate high performance computing devices into the mobile storage containers that are able to perform critical extraction, pattern, and index processing on the sensor data. The various embodiments, provide a method for the efficient physical transport of the mobile storage containers from current locations to a center analysis location for re-connecting in another peer-to-peer network for integration into a central enterprise data warehouses.

Claims

1. A method for embedding analytical software in a first mobile data storage unit in a peer-to-peer network, the first mobile data storage unit having a data processor, the method comprising:

provisioning the first mobile data storage unit at a first location, the provisioning comprising installation of an analysis software;

transporting the first mobile data storage unit to an area of operations at a second location;

connecting a plurality of second mobile data storage units with the first data storage unit in a first peer-to-peer network;

defining a custom namespace separate from at least one other namespace;

assigning the first mobile storage unit a namespace ID within the custom namespace, wherein data objects ingested into the namespace ID are assigned unique identifiers, wherein the namespace ID is persistent through disconnection and reconnection of the first mobile storage unit;

identifying one or more processing tasks to be assigned to the first mobile data storage unit, and allocating data processing to the first mobile storage unit, based on the namespace ID;

processing data in each of the mobile data storage units independently of the processing of data in the other mobile data storage units in the peer-to-peer network;

aggregating the results from the independent processing performed in each of the mobile data storage units from all the mobile data storage units within the custom namespace into a preliminary data set; and

transmitting the preliminary data set to one or more analytical environments for additional processing.

2. The method of claim 1, wherein the custom namespace is defined based on at least one criteria, the at least one criteria comprising at least one of an operating organization, a project to be associated with the namespace, or a subject matter area.

3. The method of claim 2, wherein data to be provided to the custom namespace is sorted based on the at least one criteria and the custom namespace is automatically populated with data relevant to the namespace.

4. The method of claim 1, wherein the step of connecting the plurality of second mobile data storage units with the first mobile data storage unit in the first peer-to-peer network comprises pushing one or more algorithms stored on the plurality of second mobile data storage units to the first mobile data storage unit.

5. The method of claim 4, wherein the algorithms comprise at least one of search algorithms or analysis algorithms.

6. The method of claim 1, further comprising:

pairing the first mobile storage unit with one or more mobile data storage units in the custom namespace;

replicating the data of the first mobile storage unit on the one or more paired mobile storage units; and

removing the first mobile storage unit from the peer-to-peer network;

wherein the one or more paired mobile storage units remain in operation on the peer-to-peer network.

7. The method of claim 6, further comprising:

connecting a replacement mobile data storage unit with one or more mobile data storage units in the peer-to-peer network; and

automatically replicating the data of the first mobile storage unit on the replacement mobile data storage unit.

8. The method of claim 1, wherein the one or more analytical environments perform different processing actions on the preliminary data set using different analytical techniques and approaches as required by a business process.

9. The method of claim 1, wherein data to be processed is sorted into the custom namespace based on subject matter, and wherein one or more analysis tasks precisely tailored to the subject matter is executed on the data to be processed.

10. The method of claim 1, wherein the step of processing data in each of the mobile storage units independently of the processing of data in the other mobile data storage units in the peer-to-peer network further comprises defining one or more annotations based on one or more meta-tagging standards, and assigning the one or more annotations to the data based on the one or more meta-tagging standards.

11. The method of claim 1, wherein the one or more analytical environments are at least one of another processing node on the peer-to-peer network or a separate environment on a separate network.

12. The method of claim 1, wherein at least one processor operable to execute the analysis software within the mobile storage units is dynamically allocated to the analysis software for processing data.

13. The method of claim 1, wherein an activation of the analysis software can be achieved by any user who has access to the peer-to-peer network.

14. A method for collecting, storing, and analyzing information, comprising:

provisioning a first mobile data storage unit at a first location, the first mobile data storage unit having a data processor, the provisioning comprising installation of an analysis software to ingest and process a single data type or a related family of data types in a namespace;

transporting the first mobile data storage unit to an area of operations at a second location;

connecting the first mobile data storage unit with one or more mobile data storage units in a peer-to-peer network;

collecting information in the area of operations;

wherein the first mobile storage unit is assigned a namespace ID, wherein data objects ingested into the namespace ID are assigned unique identifiers, wherein the namespace ID is persistent through disconnection and reconnection of the first mobile storage unit, and wherein the peer-to-peer network is configured to identify one or more processing tasks to be assigned to the first mobile data storage unit and allocate data processing to the first mobile storage unit based on the namespace ID; and

wherein the namespace includes a plurality of different types of storage media, the plurality of different types of storage media including the one or more mobile data storage units and at least one other type of storage media, each of the different types of storage media having a unique identifier on the namespace.

15. The method of claim 14, further comprising, following the step of connecting the first mobile storage unit with the one or more mobile data storage units in the peer-to-peer network, performing the steps of:

replicating the data of the first mobile storage unit on the one or more paired mobile storage units; and

removing the first mobile storage unit from the peer-to-peer network;

wherein the one or more paired mobile storage units remain in operation on the peer-to-peer network.

16. The method of claim 15, further comprising:

connecting a replacement mobile data storage unit with one or more mobile data storage units in the peer-to-peer network; and

automatically replicating the data of the first mobile storage unit on the replacement mobile data storage unit.

17. The method of claim 14, wherein the plurality of different types of storage media includes at least one of RAM, ROM, EEPROM, optical disc storage, magnetic disk storage, magnetic storage, or SSD storage.

18. The method of claim 14, wherein the one or more analytical environments are at least one of another processing node on the peer-to-peer network or a separate environment on a separate network.

19. The method of claim 14, wherein at least one processor operable to execute the analysis software within the mobile storage units is dynamically allocated to the analysis software for processing data.

20. The method of claim 14, wherein an activation of the analysis software can be achieved by any user who has access to the peer-to-peer network.