Method And Process For Enabling Distributing Cache Data Sources For Query Processing And Distributed Disk Caching Of Large Data And Analysis Requests

• US 20140012867A1
• Filed: 07/16/2013
• Published: 01/09/2014
• Est. Priority Date: 08/27/2010
• Status: Active Grant

• Current Assignee: Pneuron Corp. (UST Global Incorporated)
• Original Assignee: Pneuron Corp. (UST Global Incorporated)
• Inventors: Moss, Simon Byford, Elkins, Elizabeth Winters, Bachelor, Douglas Wiley, Curbelo, Raul Hugh
• Granted Patent: US 9,542,408 B2
• Time in Patent Office: Days
• US Class Current: 707/756
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/067 Enterprise or organisation modelling

Abstract

A method and system for large data and distributed disk cache processing in a Pneuron platform 100. The system and method include three specific interoperable but distributed functions: the adapter/cache Pneuron 14 and distributed disk files 34, a dynamic memory mapping tree 50, and distributed disk file cleanup 28. The system allows for large data processing considerations and the ability to access and acquire information from large data files 102 and rapidly distribute and provide the information to subsequent Pneurons 104 for processing. The system also provides the ability to store large result sets, the ability to deal with sequential as well as asynchronous parallel processing, the ability to address large unstructured data; web logs, email, web pages, etc., as well as the ability to handle failures to large block processing.

Claims

1. A method for processing large data and analysis requests, said method comprising the following acts:

acquiring data in real-time from one or more application data sources by a data processing pneuron, said data processing pneuron configured for operating under control of a computer program product, said computer program product comprising a computer program;

transforming said data into self-describing ASCII disk cache files by said data processing pneuron;

distributing said disk cache files across one or more servers or virtual machines by an adaptor cache pneuron;

allowing access to said distributed disk cache files by participating applications, wherein said applications are configured to perform operations on said distributed disk data; and

removing said disk cache files automatically using a high-low disk evaluation model to remove said distributed disk cache files based on server disk utilization and automatic evaluation aging for said distributed disk cache files.

2. A system for processing large data and analysis requests, said system comprising:

a pneuron server configured for operating under control of a computer program product, said computer program product comprising a computer program, for deploying a data processing pneuron for acquiring data in real-time from one or more application data sources, said data processing pneuron for transforming said data into one or more self-describing ASCII disk cache files;

said pneuron server configured for deploying one or more adaptor cache pneuron, said adaptor cache pneuron operating under control of a computer program product, said computer program product comprising a computer program, for distributing said one or more disk cache files across one or more servers or virtual machines;

said one or more servers configured for allowing access to said distributed disk cache files by one or more distributed worker pneurons, wherein said distributed worker pneurons are configured to perform operations on said distributed disk data; and

said one or more worker pneurons configured removing said disk cache files automatically using a high-low disk evaluation model to remove said distributed disk cache files based on server disk utilization and automatic evaluation aging for said distributed disk cache files.

3. The system of claim 2 wherein said one or more worker pneurons configured removing said disk cache files are configured for removing disk cache files prior to storing disk cache files received from said adaptor pneuron.