Bandwidth saving system and method for communicating self describing messages over a network

• US 8,855,143 B1
• Filed: 04/21/2006
• Issued: 10/07/2014
• Est. Priority Date: 04/21/2005
• Status: Active Grant

• Current Assignee: Anthony Oliveto
• Original Assignee: Anthony Oliveto
• Inventors: Acampora, Joseph
• Primary Examiner(s): ROSE, KERRI M
• Application Number:
  US11/408,448
• Time in Patent Office: 3,091 Days
• Field of Search: 370/477, 370/468, 370/252, 709/247, 709/236, 709/237, 708/203, 707/693
• US Class Current: 370/477
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 69/321 Interlayer communication protocols or service data unit [SDU] definitions; Interfaces between layers

Abstract

The enclosed system and method provides a means by which metadata-encoded or otherwise self-describing messages can be communicated, stored or otherwise utilized at a greatly reduced message size. This is accomplished for a given message through decomposing it into message metadata and message data, using a hash of the message metadata along with the message data to represent the message while in its decomposed form absent of original message metadata, and subsequently recomposing the message from the message metadata and message data using the hash as a reference. The method disclosed herein exploits detected repetition of sets of metadata, deriving benefit each time a hash is communicated, stored or otherwise used in lieu of a set of metadata, the hash being smaller in size than the set of metadata itself.

Claims

1. A computer-implemented system containing a non-transitory computer readable medium coupled to a processor, the processor operable to perform optimizing computer network bandwidth by reducing the necessary bandwidth for metadata-encoded messages sent between and amongst nodes by replacing the metadata of a signal with a unique hash of said metadata

comprising:

a metadata decompose negotiate recompose operation comprising;

(a) a processor performing a decomposition of a self-describing message into at least two constituents, a set of metadata and a set of kernel data;

(b) a processor performing a generation of a metadata hash derived from said set of metadata, said metadata hash acting as a compact replacement for said set of metadata through communication;

implementing as a compared representative for a complete identification of identical sets of metadata;

(c) a processor performing further analysis of a repetition of usage of any particular set of metadata over time;

(d) a processor performing further negotiation between a network node sending a self-describing message and a network node receiving said message, and subsequent sending and receiving of said message, at said sender and said recipient respectively, in its entirety or in one or more parts of said set of metadata, said kernel data, said metadata hash; and

(e) a processor performing a recomposition of a self-describing message to an initial form prior to said decomposition;

further an operation comprising a handshaking process by which a sending node signals to employ said metadata decompose negotiate recompose operation in the conversation, and by which a receiving node signals capability of using the method, and a subsequent initiation and use of the metadata decompose negotiate recompose operation comprising a storing of metadata hashes;

implementing said metadata decompose negotiate recompose operation;

an operation determining the presence of metadata repetition between said sending node and said receiving node; and

alternatively an operation detecting the presence of metadata repetition from said sending node irrespective of a receiving node.

2. The computer implemented system comprising an operation of claim 1 wherein the sending node is a non endpoint node, the receiving node is a non endpoint node and further wherein said sending endpoint node and said receiving node communicate to establish a virtual tunnel for a conversation for nodes other then said sending and said receiving nodes.

3. The computer-implemented system of claim 2 further comprising an operation wherein one of the communicating nodes is a cell phone.

4. The computer-implemented system of claim 1 comprising an operation wherein one of the communicating nodes is a multiplayer gaming system node.

5. The computer-implemented system of claim 1 further comprising an operation of a canonicalization of the metadata structure prior to creating a hash.

6. The computer-implemented system of claim 1 further comprising an operation wherein said

metadata decompose negotiate recompose renegotiate operation

is implemented using a program executed on a processor other than a central processing unit of the computing node or a networking device node.

7. A non transitory computer readable medium coupled to a processor, the processor operable to perform for optimizing computer network bandwidth by reducing the necessary bandwidth for metadata-encoded messages sent between and amongst nodes by replacing the metadata of a signal with a unique hash of said metadata of claim 1

further comprising:

a storage of said self-describing message.

8. The system of claim 1 further wherein

said self-describing message comprises a metadata structure canonicalized prior to creating a metadata hash derived from said set of metadata;

and

said self-describing message is stored comprising

a set of kernel data;

a set of metadata;

a metadata hash derived from said set of metadata.

9. A computer-implemented method for optimizing computer network bandwidth reducing the necessary bandwidth for signaling comprising

at least one or more

metadata-encoded messages comprising a set of metadata and a kernal data sent between and amongst nodes in a network comprising

said nodes by a system generated substitution of metadata with at least one of a signal comprising a hash

wherein said hash comprises

a substitution signal for a set of metadata decomposed from said metadata encoded messages

for a set of metadata including one or more placeholders decomposed from metadata encoded messages;

wherein said placeholders substitute for a location of kernel data of metadata encoded messages;

and

a handshaking process signaling comprising

a sending entity node signaling requests to employ metadata decompose negotiate recompose in a conversation on said network, and by which

a receiving entity signals a participation in said handshaking process;

a sending entity node signaling initiation of said method; and

a fallback to standard communications to recover comprising from any failure of said metadata-encoded messages;

a failure of said handshaking process comprising

either a sending node or a receiving node with

a detecting of a timeout;

an error message, a handshaking process invalidity.

10. The system of claim 9 further wherein

said handshaking process signaling comprising

a receiving entity node signaling requests to employ metadata decompose negotiate recompose in a conversation on said network, and by which

a sending entity signals a participation in said handshaking process;

a receiving entity node a signaling initiation of said method; and

a fallback to a standard communications to recover from any failure of the method comprising either sending or receiving node detecting a timeout;

an error message, a handshaking process invalidity.

11. A non transitory computer coupled to a processor programmed to generate efficient metadata comprising:

a metadata generated by a system comprising;

a compression source code segment comprising a self-describing source code;

a computer data metadata further comprising

a system messaging as a D-IDS, Distributed Intrusion Detection System with two message recipients comprising at least one of an executable to detect network intrusions, an IDMEF Intrusion Detection Message Exchange Format, or a D-IDS Distributed Intrusion Detection Systems network architectures membership achieving bandwidth savings for message volume, with limited impacts during messaging communicator negotiation initiation

further comprising

a system messaging as a D-IDS, with at least two message recipients, comprising SNORT™

, IDMEF, D-IDS network architectures

; and

said system achieving limited network latency for a bloated message, with limited impacts of latency during

messaging negotiation phase messaging networks said messaging networks further comprising;

a messaging to and from metadata of

wireless local area network, a network interface controller local area network adapter central processor unit computer node, an internal network interface card on a computer node, a computer with a personal computer card adapter;

an Ethernet, a Token ring, a structured wiring system, a browser, a Rich Site Summary feed syndication service;

a Mobile Telephone Switching Office, a thin client, a peer to peer network with a name server, a peer to peer network without a name server, a System area network, a Metropolitan Area Network, a circuit switched digital network, an Integrated Services for Digital Network network, a packet switched network, an Asynchronous Transfer Mode switch, a central processor unit with a Browser plug in, a Web server plug in module, a web services enabled central processor unit, a home network, a financial transactions network, a radio network, a modem central processor unit circuit, or a medical data service.

12. A method of compressing data for network messaging, comprising:

obtaining a metadata encoded message;

decomposing an initial set of signals to obtain a hash for each signal in the initial set of relevant signals

separating out a set of metadata;

determining a suitability of sending messaging based on repetition of a particular set of metadata in said metadata encoded message comprising data structures;

performing a calculation on said set of metadata comprising

at least one of a cryptographic hashing function further comprising at least a hash value based on an input further comprising wherein said input is said set of metadata

performing a calculation on said set of metadata signals to obtain a hash;

comparing the hash for at least two of the messages compared message-by-message in said initial set of relevant signals;

and transmitting said set of relevant signals based on the comparison of the hash.

13. A system comprising:

a server connected to a network, the server securely receiving hash comparisons from users via the network, the server including;

at

least one processor;

a storage memory comprising of

database of a network corpus of

signals;

a memory operatively coupled to the processor, the memory storing program instructions that when executed by the processor, causes the processor to;

generate an initial set of relevant signals from a network corpus of signals comprising data structures;

obtain a hash based on a set of metadata for each

signal in the initial set of relevant signals;

compare the hash for at least two of metadata encoded messages in the initial set of relevant signals; and

transmit the set of relevant signals based on the comparison of the hash.

14. The system of claim 13 further wherein implementing at an application layer calling routines from a-metadata decompose negotiate recompose library.

15. The system of claim 13 further wherein perform each of said signaling functions independently as requested by the application;

receiving a request from the application in the form of a message and a recipient'"'"'s network address, to perform said metadata decompose negotiate recompose protocols.

16. The system of claim 13 further wherein a status of the communication is signaled to the application.

17. The system of claim 13 further wherein said system at a transport layer implements with the application independent of the use of said—

metadata decompose negotiate recompose further comprising an metadata decompose negotiate recompose protocol driver installed with an operating system;

a virtual machine, and interrelates to networking components of said OS with an metadata decompose negotiate recompose protocol driver listening for metadata-encoded messages as traffic is passed down the network stack.

18. A non transitory computer readable medium storing instructions for causing at least one processor to perform a method

obtaining an initial set of relevant messages comprising metadata from a network corpus of messages comprising data structures;

compacting the initial set of messages to obtain a hash for each signal in the initial set of messages comprising metadata;

comparing the hash for at least two of the data structures in the initial set of relevant messages comprising metadata;

and transmitting the set of relevant messages comprising at least one of a metadata or a hash substituted for set of relevant messages metadata based on the comparison of said hash.

19. A computer-implemented method for optimizing computer network bandwidth by reducing the necessary bandwidth for metadata-encoded messages sent between and amongst nodes by replacing the metadata of a signal with a unique hash of set of metadata of said signal;

said method comprising;

participating in a messaging system comprising transmission of self describing messages;

decomposing an initial message comprising a set of initial data excluding metadata;

a set of metadata; and

a hash product of decomposition of said set of metadata;

sending said hash of said set of metadata and said set of initial data excluding metadata to a participating recipient node which omits unnecessary repetition of said set of metadata;

recomposing from said hash of said set of metadata and the set of initial data into said initial message;

said recomposing comprising

using said hash of said set of metadata as a reference to said set of metadata at recipient node;

substituting in said reference set of metadata;

said set of initial data corresponding to a placeholder signal in said reference set of metadata.

20. The computer implemented method of claim 19 further wherein comprising the steps of:

receiving a message said message comprising

invoking a method for messaging comprising

receiving a self-describing message from a signal;

said self describing message

comprising a set of data, comprising zero or more data items,

a metadata item

comprising at least one of a starting tag, ending tag;

a metadata versioning element;

a metadata for namespace;

customizable elements;

a kernel data decomposing at least one of several self-describing messages into constituents comprising a metadata constituent and the data constituent itself;

generating a unique hash based on said metadata item;

generating a control signal representing a placeholder substituting for the data items metadata of said initial message;

generating a delimiting signal for parsing of data items within said data constituent.

21. The computer implemented method of claim 20 further wherein comprising the steps of:

analyzing occurrences of repetition of usage of any particular whole structure of metadata over time;

negotiating with a recipient for transmission of said hash and said data constituent itself as substitute for initial message;

said transmission further comprising said initial message;

said hash;

said set of metadata;

said delimited data;

a customizable element;

transmitting to a separate node a signal comprising said unique hash;

said data

comprising data parsed from said initial message, at least one of a delimiter of data;

field offsets, means for parsing said data,

said metadata, said control signal system generated placeholder symbol to identify data location in initial message;

means for identifying fields in said data;

storing at least one of a decomposed message comprising the metadata;

hash;

one or more occurrences of the hash with data;

said decomposed message data comprising signaling comprising delimiters, an offset, a mapping;

recomposing said initial message from said hash;

said delimiters of data;

a field offsets;

a mask;

a customizable marker.

22. A non transitory computer readable medium coupled to a processor, the processor operable to perform optimizing computer network bandwidth by reducing the necessary bandwidth for metadata-encoded messages sent between and amongst nodes by replacing the metadata of a signal with a unique hash of said metadata

comprising:

a tight coupling metadata decompose negotiate recompose comprising;

(a) a decomposition of a self-describing message into at least two constituents, a set of metadata and a set of kernel data;

(b) a generation of a metadata hash derived from said set of metadata, said metadata hash acting as a compact replacement for said set of metadata through communication;

storage of said self-describing message,

implementing as a compared representative for a complete identification of identical sets of metadata;

(c) further analysis of a repetition of usage of any particular set of metadata over time;

(d) further negotiation between a network node sending a self-describing message and a network node receiving said message, and subsequent sending and receiving of said message, at said sender and said recipient respectively, in its entirety or in one or more parts of said set of metadata, said kernel data, said metadata hash; and

(e) a recomposition of a self-describing message to an initial form prior to said decomposition;

further comprising a handshaking process by which a sending node signals to employ said metadata decompose negotiate recompose operation in the conversation, and by which a receiving node signals capability of using the method, and a subsequent initiation and use of the -metadata decompose negotiate recompose operation comprising a storing of metadata hashes;

analyzing need of implementing said tight coupling metadata decompose negotiate recompose;

implementing said tight coupling metadata decompose negotiate recompose;

determining the presence of metadata repetition between said sending node and said receiving node; and

alternatively detecting the presence of metadata repetition from said sending node irrespective of a receiving node;

maintaining at a sending node a history of communications such that existence of said set of metadata at said receiving node can be predicted;

said set of metadata at said receiving node is predicted, sending

said hash of metadata and said set of kernel data;

a receiving node buffering said metadata hash and said

set of kernel data where said receiving node has no corresponding

set of metadata based on said metadata hash, anticipating, a receipt

of said set of metadata from said sending node and;

a receiving node communicating to said sending node a request for said set of metadata based on said hash of said set of metadata.

23. One or more non transitory computer processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method for optimizing computer network bandwidth by reducing the necessary bandwidth for metadata-encoded messages sent between and amongst nodes by replacing the metadata of a signal with a unique hash of set of metadata of said signal, the method comprising the steps of:

participating in a messaging system comprising transmission of self describing messages;

decomposing an initial message comprising a set of initial data excluding metadata;

a set of metadata; and

a hash product of decomposition of said set of metadata;

sending said hash of said set of metadata and said set of initial data excluding metadata to a participating recipient node which omits unnecessary repetition of said set of metadata;

recomposing from said hash of said set of metadata and the set of initial data into said initial message;

said recomposing comprising

using said hash of said set of metadata as a reference to said set of metadata at recipient node;

substituting in said reference set of metadata;

said set of initial data corresponding to a placeholder signal in said reference set of metadata.

24. One or more non transitory computer processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method for optimizing computer network bandwidth by reducing the necessary bandwidth for metadata-encoded messages sent between and amongst nodes by replacing the metadata of a signal with a unique hash of set of metadata of said signal,

the method of claim 23 further comprising

the step of:

storing of said self-describing message.