Method and process for enabling distributing cache data sources for query processing and distributed disk caching of large data and analysis requests

• US 9,542,408 B2
• Filed: 07/16/2013
• Issued: 01/10/2017
• Est. Priority Date: 08/27/2010
• Status: Active Grant

• Current Assignee: UST Global (Singapore) Pte Ltd. (UST Global Incorporated)
• Original Assignee: Pneuron Corp. (UST Global Incorporated)
• Inventors: Moss, Simon Byford, Elkins, Elizabeth Winters, Bachelor, Douglas Wiley, Curbelo, Raul Hugo, Fountain, Thomas C.
• Primary Examiner(s): Onat, Umut
• Application Number:
  US13/943,187
• Publication Number:
  US 20140012867A1
• Time in Patent Office: 1,274 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/067 Enterprise or organisation modelling

Abstract

A method and system for large data and distributed disk cache processing in a Pneuron platform 100. The system and method include three specific interoperable but distributed functions: the adapter/cache Pneuron 14 and distributed disk files 34, a dynamic memory mapping tree 50, and distributed disk file cleanup 28. The system allows for large data processing considerations and the ability to access and acquire information from large data sets 102a, 102b and rapidly distribute and provide the information to subsequent Pneurons 104 for processing. The system also provides the ability to store large result sets, the ability to deal with sequential as well as asynchronous parallel processing, the ability to address large unstructured data; web logs, email, web pages, etc., as well as the ability to handle failures to large block processing.

Claims

1. A method for distributed processing of data, comprising the steps of:

acquiring, by an adaptor neuron executing on a processor, data from a data source;

creating, by the adaptor neuron on a first computer, a self-describing disk cache file based at least in part upon the data acquired from the data source;

creating, by the adaptor neuron, a copy of the adaptor neuron and a copy of the self-describing disk cache file on a second computer;

permitting access to the self-describing disk cache file through the adaptor neuron by a first participating application that is configured to perform a first data processing operation;

permitting access to the copy of the self-describing disk cache file through the copy of the adaptor neuron by a second participating action that is configured to perform a second data processing operation;

maintaining, by the adaptor neuron and the copy of the adaptor neuron, a memory tree map of change history to both the self-describing disk cache file and the copy of the self-describing disk cache file; and

retrieving, by the copy of the adaptor neuron, in response to a request for information in the copy of the self-describing disk cache file at a point in the change history, the information from the copy of the self-describing disk cache file at memory location in the memory tree map corresponding to the point in the change history.

2. The method of claim 1, wherein at least one of the first computer and the second computer is a virtual machine.

3. The method of claim 1 wherein the access to the self-describing disk cache file and the copy of the self-describing disk cache file occurs simultaneously to allow multiple simultaneous operations on the information contained in the self-describing disk cache file and the copy of the self-describing disk cache file.

4. The method of claim 3, wherein at least one of the first computer and the second computer is a virtual machine.

5. The method of claim 1, further comprising the step of removing the self-describing disk cache file based at least in part on a high-low disk evaluation model.

6. The method of claim 5, wherein at least one of the first computer and the second computer is a virtual machine.

7. The method of claim 1, further comprising the step of removing the self-describing disk cache file based at least in part on an aging evaluation.

8. The method of claim 7, wherein at least one of the first computer and the second computer is a virtual machine.