Realtime processing of streaming data

• US 9,652,312 B2
• Filed: 07/01/2015
• Issued: 05/16/2017
• Est. Priority Date: 07/03/2014
• Status: Active Grant

• Current Assignee: Fisheye Products LLC
• Original Assignee: Fisheye Products LLC
• Inventors: Selig, IV, Ernest Theodore, Ackroyd, Michael Harold
• Primary Examiner(s): Dao, Tuan
• Assistant Examiner(s): Wood, William C
• Application Number:
  US14/789,197
• Publication Number:
  US 20160004578A1
• Time in Patent Office: 685 Days
• Field of Search: None
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/541 via adapters, e.g. between incompatible applications
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues

Abstract

The invention described here is intended for enhancing the technology domain of real-time and high-performance distributed computing. This invention provides a connotative and intuitive grammar that allows users to define how data is to be automatically encoded/decoded for transport between computing systems. This capability eliminates the need for hand-crafting custom solutions for every combination of platform and transport medium. This is a software framework that can serve as a basis for real-time capture, distribution, and analysis of large volumes and variety of data moving at rapid or real-time velocity. It can be configured as-is or can be extended as a framework to filter-and-extract data from a system for distribution to other systems (including other instances of the framework). Users control all features for capture, filtering, distribution, analysis, and visualization by configuration files (as opposed to software programming) that are read at program startup. It enables large scalable computation of high velocity data over distributed heterogeneous platforms. As compared with conventional approaches to data capture which extract data in proprietary formats and rely upon post-run standalone analysis programs in non-real-time, this invention also allows data streaming in real-time to an open range of analysis and visualization tools. Data treatment options are specified via end-user configuration files as opposed to hard-coding software revisions.

Claims

1. A computer-implemented method for transferring data from a source platform to one or more target platforms, comprising:

from a source platform and a native binary data stored in memory on the source platform, determining schema for an application native to the source platform based on at least one of;

software code, files related to the application, a compiler, and platform information that characterizes data layout and computational interpretation of the native binary data on the source platform, wherein the platform information includes encoding policy settings;

communicatively coupling the source platform to the one or more target platforms enabling the one or more target platforms to access a message including the determined schema and platform information of the source platform;

converting the determined schema to a form useable by the one or more target platforms, the converted schema enabling the one or more target platforms to synthesize meaning from the native binary data of the source platform without converting the native binary data; and

on each target platform, before converting the native binary data, utilizing the converted schema and platform information to synthesize meaning from the native binary data by identifying data fields within the native binary data from the source platform, selecting one or more of the identified data fields, and converting data of the one or more selected data fields to a data format useable by operations on the target platform.

2. The method of claim 1, wherein the source platform is the target platform.

3. The method of claim 1, wherein the source platform and the one or more target platforms are heterogeneous.

4. The method of claim 1, further comprising, distinguishing, using the encoded policy settings, data encodings in native binary form on the source platform from data encodings in native binary form on the one or more target platforms.

5. The method of claim 1, further including utilizing the the identified data fields for operations on the one or more target platforms.

6. The method of claim 1, wherein the schema is metadata created by:

(a) accounting for structure, terminology, layout, and semantics of native binary data on the source platform and formulating a metadata representation, and (b) maintaining the formulated metadata representation in a sharable and distributable form.

7. The method of claim 1, wherein the source platform and the one or more target platforms are capable of being operated concurrently both among the target platforms and with the source platform.

8. The method of claim 1 wherein converting the determined schema further includes:

based on the encoded policy settings, mapping a structural form of the native binary data on the source platform to an operational form in the target platform; and

applying said mapping from the native binary data structural form on the source platform to the operational form in the target platform.

9. The method of claim 1 further comprising:

devising ontology representations that infer meaning from patterns of plural schemas and established ontologies;

recognizing patterns of data in the determined ontology representations that match the established ontologies; and

synthesizing information based on matched ontologies.

10. The method of claim 9 wherein devising ontology representations includes:

identifying schemas and ontologies associated with the established ontology;

mapping data fields of the identified schemas and ontologies; and

for each data field mapping, defining an expression that transforms a value of the data field in data of the source platform into one or more data field values on the target platform.

11. The method of claim 9 wherein the determined ontology representations are maintained in a shareable and distributable form.

12. A system for ontology-based concurrent information synthesis, comprising:

a) a source platform which includes;

i) a native binary data stored in memory; and

ii) a module for determining schema for an application native to the source platform based on at least one of;

software code, files related to the application, a compiler, and platform information that characterize data layout and computational interpretation of the native binary data on the source platform, wherein the platform information includes encoding policy settings;

b) one or more target platforms;

c) a network configured to communicatively coupling the source platform to the one or more target platforms, enabling the one or more target platforms to access a message including the determined schema and platform information of the source platform;

d) a module for converting the determined schema to a form useable by the one or more target platforms, the converted schema enabling the one or more target platforms to synthesize meaning from the native binary data of the source platform without converting the native binary data;

e) on each target platform, a module for synthesizing meaning from native binary data configured to;

(1) utilize the converted schema and platform information, before converting the native binary data, to synthesize meaning from the native binary data by identifying data fields within the native binary data from the source platform, (2) select one or more of the identified data fields, and (3) convert data of the one or more selected data fields to a data format useable by operations on the target platform.

13. The system of claim 12 wherein the source platform is the target platform.

14. The system of claim 12 wherein the source platform and the one or more target platforms are heterogeneous.

15. The system of claim 12 wherein the encoding policy settings distinguish how data is encoded in native binary form on the source platform from how that data is encoded in native binary form on the one or more target platforms.

16. The system of claim 12 wherein the module for synthesizing meaning from native binary data is further configured to enable operations on the target platform to access the identified data fields.

17. The system of claim 12 wherein the schema is metadata created by a metadata generation module that accounts for structure, terminology, layout, and semantics of native binary data on the source platform, formulates a metadata representation, and maintains the formulated metadata representation in a sharable and distributable form.

18. The system of claim 12 wherein the source platform and the one or more target platforms are operated concurrently both among the target platforms and with the source platform.

19. The system of claim 12 wherein the module for extracting establishes a mapping from structural form of the native binary data on the source platform to an operational form in the target platform, based on the encoding policy settings, and applies said mapping from the native binary data structural form on the source platform to the operational form in the target platform.

20. The system of claim 12 further comprising:

a module for devising ontology representations, wherein the module for devising ontology representations infers meaning from patterns of plural schemas and established ontologies, recognizes patterns of data based on the devised ontology representations that match established ontologies, synthesizes information based on matched ontologies.

21. The system of claim 20 wherein the module for devising ontology representations additionally identifies schemas and ontologies associated with the established ontologies, maps data fields of the identified and ontologies, and for each data field mapping, defines an expression that transforms a value of the data field in data of the source platform into one or more data field values on the target platform.

22. The method of claim 20 wherein the devised ontology representations are maintained in a shareable and distributable form.