Network architecture and methods for transparent on-line cross-sessional encoding and transport of network communications data

• US 7,336,682 B2
• Filed: 07/25/2001
• Issued: 02/26/2008
• Est. Priority Date: 07/25/2000
• Status: Active Grant

• Current Assignee: Riverbed Technology, LLC (Riverbed Technology Incorporated)
• Original Assignee: Juniper Networks Incorporated
• Inventors: Singh, Amit P
• Primary Examiner(s): Pham; Chi
• Assistant Examiner(s): MEW, KEVIN D
• Application Number:
  US09/915,939
• Publication Number:
  US 20020071438A1
• Time in Patent Office: 2,407 Days
• Field of Search: 370/469, 370/471, 370/475, 370/393, 341/4, 341/13, 341/51
• US Class Current: 370/474
• CPC Class Codes: H04L 12/4633 Interconnection of networks using encapsulation techniques, e.g. tunneling
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation

Abstract

A computer implemented method and system for transferring data packets includes intercepting a stream of data packets at the connectionless network layer from a client or server, encoding and encapsulating the data packets, transmitting the encoded data packets, decoding and decapsulating the data packets, and injecting the decoded and decapsulated data packets into the connectionless network layer at a client or server. The encoding of the packets is based on detecting repetitions that could exist anywhere in the data stream. The repetitions are not necessarily within a single packet but could occur across multiple packets and multiple sessions. One of the encoding algorithms includes comparing the payloads of packets with identical signatures and transmitting the full packet only when the payload had not previously been sent.

Claims

1. A system for encoding one or more repetitive data blocks in data communicated over a network comprising:

an encoder module, coupled in the network, the encoder module intercepting data communicated from a server to a client device, the encoder module extracting data blocks from different communication sessions from the intercepted data, each extracted block having a destination address of the client device, the encoder module determining whether the client device is supported for decoding by at least one corresponding decoder module based on the destination address of the client device, and the encoder module passing through the data when the destination address of the client device is not supported for decoding by a decoder module; and

a memory, accessible to the encoder module, for storing the contents of one or more data blocks previously transmitted by the encoder module wherein the encoder module determines whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block,

wherein the memory comprises a least recently used data structure for storing one or more previously transmitted unique data blocks, said least recently used data structure having a maximum capacity and each of the previously transmitted unique data blocks having a unique identifier and a position in an order of most recently used to least recently used of the one or more stored blocks,

wherein responsive to said match, the encoder module encodes the respective extracted data block, transmits the respective extracted data block in encoded form to the at least one corresponding decoder module and associates the previously transmitted data block having the matching contents with the position in the least recently used data structure indicating the most recently used previously transmitted data block, and

wherein responsive to no match, the encoder module transmits the respective extracted data block in intercepted form to the at least one corresponding decoder module.

2. The system of claim 1 wherein the encoded form of the respective extracted data block is transparent to one or more nodes in the network.

3. The system of claim 1 wherein the encoder module is coupled via a switch in a physical connection between two nodes of the network, responsive to a first configuration of the switch, the encoder module processing data that traverse the physical connection between these two nodes, and responsive to a second configuration of the switch, the data bypassing the encoder module.

4. The system of claim 1 wherein the encoder module operates in a node in the network and decides a route for the respective extracted data block to the at least one corresponding decoder module supporting its destination address.

5. The system of claim 1 wherein the encoder module, responsive to a match in contents, transmits art indicator, identifying that the contents of a respective data block have been previously transmitted.

6. The system of claim 5 wherein the indicator is a special symbol.

7. The system of claim 5 wherein the indicator is an extra header.

8. The system of claim 1 wherein at least one respective data block is a pacter payload.

9. The system of claim 1 wherein at least one respective data block is a portion of a packet payload.

10. The system of claim 1 wherein the encoder module encodes at least one extracted data block using a synchronization mechanism for verifying the identification of the one or more previously transmitted data blocks with the at least one corresponding decoder module supporting the destination address of the extracted data block.

11. The system of claim 10 wherein the synchronization mechanism is an explicit synchronization mechanism.

12. The system of claim 10 wherein the synchronization mechanism is an implicit synchronization mechanism.

13. The system of claim 10 wherein the implicit synchronization mechanism is a reliable network transport protocol.

14. The system of claim 1 further comprising the encoder module, responsive to no match between the contents of the respective extracted data block and the contents of at least one of the previously transmitted blocks, determining whether to delete at least one of the previously transmitted data blocks in the memory, and storing the respective extracted data block in intercepted form in the memory.

21. The system of claim 1 wherein the memory stores a data structure for associating a signature with one or more of the previously transmitted data blocks, and wherein the encoder module computes a signature for the respective extracted data block, compares the computed signature with at Least one signature associated with the one or more previously transmitted data blocks, and responsive to a match in signature, selecting the one or more previously transmitted data blocks having the match in signature for content comparison with the respective extracted data block.

22. The system of claim 21 wherein the data structure for associating a signature with one or more of the previously transmitted data blocks is a hash table having one or more bins.

25. The system of claim 1 further comprising an encapsulation module for encapsulating the extracted data blocks for transport over the network.

26. The system of claim 25 wherein at least one of the extracted data blocks is included in a packet, and the packet is encapsulated as one packet.

27. The system of claim 25 wherein at least one of the extracted data blocks is included in a packet and the packet is encapsulated with at least one other packet in an outgoing packet for transmission.

28. The system of claim 25 wherein the encapsulation of at least one of the extracted data blocks is transparent to one or more nodes in the network.

29. The system of claim 25 wherein the encapsulation module comprises a timer mechanism for ensuring that the at least one extracted data block is held in a buffer coupled to the encapsulation module for no more than a pre-determined maximum time before being transmitted.

30. The system of claim 27 wherein the encapsulation module comprises a timer mechanism for ensuring that the at least one extracted data block is held in a buffer coupled to the encapsulation module for no more than a pre-determined maximum time before being transmitted.

31. The system of claim 25 wherein the encoder module encodes at least one data block at a first layer of a model describing the flow of data across a network and the encapsulation module encapsulates the at least one extracted data block at a second layer of the model.

32. The system of claim 31 wherein the first and second layers are at the same layer of the model.

33. The system of claim 32 wherein one of the layers is a connection-oriented layer and the other layer is a connectionless layer.

15. A system for encoding one or more repetitive data blocks in data communicated over a network comprising:

an encoder module, coupled in the network, the encoder module intercepting data communicated from at least one server to at least one client device, the encoder module extracting data blocks from different communication sessions from the intercepted data, each extracted block having a destination address of the client device to which the extracted data block is destined, the encoder module determining whether the client device is supported for decoding by at least one corresponding decoder module based on the destination address of the client device, and the encoder module passing through data when the destination address of the client device is not supported for decoding by a decoder module; and

a memory, accessible to the encoder module, having a least recently used data structure for storing the contents of one or more unique data blocks previously transmitted by the encoder module, said least recently used data structure having a maximum capacity and each of the previously transmitted unique data blocks having a unique identifier and a position in an order of most recently used to least recently used of the one or more stored blocks,

wherein the encoder module determines whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block,

wherein responsive to said match, die encoder module encodes the respective extracted data block and transmits the respective extracted data block in encoded form to the at least one corresponding decoder module, and

wherein responsive to no match, the encoder module transmits the respective extracted data block in intercepted form to the at least one corresponding decoder module, stores the extracted data block in the least recently used data structure, and associates the position of most recently used with the extracted data block.

16. The system of claim 15 wherein, responsive to the least recently used data structure being at the maximum capacity, the encoder module deletes the previously transmitted data block having the order position of the least recently used data block.

17. A system for encoding one or more repetitive data blocks in data communicated over a network comprising:

an encoder module, coupled in the network, the encoder module intercepting the data, the encoder module extracting data blocks from different communication sessions from the intercepted data;

a memory, accessible to the encoder module, having a least recently used data structure for storing the contents of one or more unique data blocks previously transmitted by the encoder module, said least recently used data structure having a maximum capacity and each of the previously transmitted unique data blocks having a unique identifier and a position in an order of mast recently used to least recently used of the one or more stored blocks,

wherein, responsive to the least recently used data structure being at the maximum capacity, the encoder module deletes the previously transmitted data block having the order position of the least recently used data block,

wherein the encoder module determines whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block,

wherein responsive to said match, the encoder module encodes the respective extracted data block and transmits the respective extracted data block in encoded form to at least one corresponding decoder module, and

wherein responsive to no match, the encoder module transmits the respective extracted data block in intercepted form to the at least one corresponding decoder module, stores the extracted data block in the least recently used data structure, and associates the position of most recently used with the extracted data block; and

a synchronization mechanism including a same size for the least recently used data structure as the size of a second least recently used data structure accessible by the corresponding decoder module receiving the respective extracted data block, and a reliable network transport protocol being used for the transmission of the respective extracted data block.

18. A system for encoding one or more repetitive data blocks in data communicated over a network comprising:

an encoder module, coupled in the network, the encoder module intercepting the data, the encoder module extracting data blocks from different communication sessions from the intercepted data;

a memory, accessible to the encoder module, having a least recently used data structure for storing the contents of one or more unique data blocks previously transmitted by the encoder module, said least recently used data structure having a maximum capacity and each of the previously transmitted unique data blocks having a unique identifier and a position in an order of most recently used to least recently used of the one or more stored blocks,

wherein, responsive to the least recently used data structure being at the maximum capacity, the encoder module deletes the previously transmitted data block having the order position of the least recently used data block,

wherein the encoder module determines whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block,

wherein responsive to said match, the encoder module encodes the respective extracted data block and transmits the respective extracted data block in encoded form to at least one corresponding decoder module, and

wherein responsive to no match, the encoder module transmits the respective extracted data block in intercepted form to the at least one corresponding decoder module, stores the extracted data block in the least recently used data structure, and associates the position of most recently used with the extracted data block; and

a synchronization mechanism including an indicator for each transmitted extracted data block indicating whether the data block has been associated with the least recently used data structure and an installed flag associated with each of the previously transmitted data blocks in the least recently used data structure, the installed flag indicating whether the associated data block has been stored in a second least recently used data structure accessible by the corresponding decoder module that has received the respective extracted data block.

19. The system of claim 18 wherein the installed flag is updated after a fixed amount of time after transmission of the data block.

20. The system of claim 18 wherein the installed flag is updated responsive to a notification from the corresponding decoder module that the associated data block has been stored in the second least recently used data structure.

23. A system for encoding one or more repetitive data blocks in data communicated over a network comprising:

an encoder module intercepting the data and extracting data blocks from different communication sessions from the intercepted data, each extracted block having a destination address of a client device to which the extracted data block is destined and that is supported for decoding by at least one corresponding decoder module, and the encoder module passing through data not having a supported destination address;

a memory, accessible to the encoder module, for storing the contents of one or more data blocks previously transmitted by the encoder module; and

a hash table having one or more bins for associating a signature with one or more of the previously transmitted data blocks,

wherein the encoder module determines whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block by computing a signature for the respective extracted data block and comparing the computed signature with at least one signature associated with the one or more previously transmitted data blocks, the computed signature value being less than the number of hash table bins,

wherein, responsive to a match in signature, the encoder selects the one or more previously transmitted data blocks having the match in signature for content comparison with the respective extracted data block, encodes the respective extracted data block and transmits the respective extracted data block in encoded form to the at least one corresponding decoder module, and

wherein, responsive to no match, the encoder module transmits the respective extracted data block in intercepted form to the at least one corresponding decoder module.

24. A system for encoding one or more repetitive data blocks in data communicated over a network comprising:

an encoder module intercepting the data and extracting data blocks from different communication sessions from the intercepted data, each extracted block having a destination address of a client device to which the extracted data block is destined and that is supported for decoding by at least one corresponding decoder module, and the encoder module passing through data not having a supported destination address;

a memory, accessible to the encoder module, for storing the contents of one or more data blocks previously transmitted by the encoder module; and

a hash table having one or more bins for associating a signature with one or more of the previously transmitted data blocks,

wherein the encoder module determines whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block by computing a signature for the respective extracted data block and comparing the computed signature with at least one signature associated with the one or more previously transmitted data blocks, the computed signature value being computed as a modulo of the number of bins,

wherein, responsive to a match in signature, the encoder selects the one or more previously transmitted data blocks baying the match in signature for content comparison with the respective extracted data block, encodes the respective extracted data block and transmits the respective extracted data block in encoded form to the at least one corresponding decoder module, and

wherein, responsive to no match, the encoder module transmits the respective extracted data block in intercepted form to the at least one corresponding decoder module.

34. A system for encoding one or more repetitive data blocks in data communicated over a network comprising:

an encoder module intercepting data communicated from a server to a client device and extracting data blocks from different communication sessions from the intercepted data, each extracted block having a destination address of the client device, the encoder module determining whether the client device is supported for decoding by at least one corresponding decoder module based on the destination address of the client device, and the encoder module passing through the data when the destination address of the client device is not supported for decoding by a decoder module; and

a memory, accessible to the encoder module, for storing the contents of one or more data blocks previously transmitted by the encoder module,

wherein the encoder module determines whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block,

wherein responsive to said match, the encoder module encodes the respective extracted data block and transmits the respective extracted data block in encoded form to the at least one corresponding decoder module,

wherein responsive to no match, the encoder module transmits the respective extracted data block in intercepted form to the at least one corresponding decoder module, and

wherein the encoder module receives routing information in accordance with a routing protocol over the network from each of one or mote corresponding decoder modules with which it communicates and determines the one or more addresses supported by each respective decoder module from the routing information, wherein the routing information includes network topology information for the network.

35. The system of claim 34 wherein the encoder module determines the network topology information based upon the routing information from the one or more decoder modules.

36. The system of claim 35 wherein, responsive to multiple decoder modules in the network supporting the same address, the encoder module determines a destination decoder module for one or more extracted data blocks based upon network topology information and routing criteria.

37. The system of claim 35 wherein the routing criteria includes routing cost.

38. The system of claim 35 wherein the routing criteria includes routing distance.

39. A system for decoding one or more repetitive data blocks in data communicated over a network comprising:

a decoder module being coupled in the network, the decoder module transmitting routing information over the network to a corresponding encoder module in accordance with a routing protocol, the routing information including network topology information for the network and identifying one or more addresses of client devices that the decoder module supports to control selective encoding of data at the encoder module based on the client devices supported by the decoder module by determining with the encoder module whether client devices are supported for decoding by at least one corresponding decoder module based on destination addresses of the client devices and passing through the data when the destination addresses of the client devices are not supported by decoding by a decoder module, the decoder module receiving data blocks for different communication sessions from the corresponding encoder module, each of the data blocks having a destination address of a client device to which the extracted data block is destined; and

a memory accessible to the decoder module for storing the contents of one or more data blocks previously received from the corresponding encoder module wherein the decoder module determines whether the contents of each of the received data blocks is in encoded form,

wherein the memory stores the previously received data blocks in a least recently used data structure for storing one or more previously received unique data blocks, the least recently used data structure having a maximum capacity and each of the previously received unique data blocks having a unique identifier and a position in an order of most recently used to least recently used of the one or more previously received blocks,

wherein responsive to the respective data block being in encoded form, the decoder module selects the contents of a matching previously received block as the contents for the respective encoded block and associates the previously received data block having the matching contents with the position in the least recently used data structure indicating the most recently used previously received data, and

wherein responsive to the data block being encoded, the decoder module stores the contents of the respective received data block as a previously received data block.

40. The system of claim 39 wherein the decoder module is coupled via a switch in a physical connection between two nodes of the network, responsive to a first configuration of the switch, the encoder module processing data that traverse the physical connection between these two nodes, and responsive to a second configuration of the switch, the data bypassing the decoder module.

41. The system of claim 39 wherein the decoder module operates in a node in the network and decides a route for a decoded data block to its destination address.

42. The system of claim 39 wherein the decoder module receives an indicator for identifying that the contents of at least one encoded data block have been previously transmitted to the decoder module.

43. The system of claim 42 wherein the indicator is a special symbol.

44. The system of claim 42 wherein the indicator is an extra header.

45. The system of claim 39 wherein at least one encoded data block is a packet payload.

46. The system of claim 39 wherein at least one encoded data block is a portion of a packet payload.

47. The system of claim 39 wherein the decoder module decodes at least one encoded data block using a synchronization mechanism for verifying the identification of the one or more previously received data blocks from a corresponding encoder module.

48. The system of claim 47 wherein the synchronization mechanism is an explicit synchronization mechanism.

49. The system of claim 47 wherein the synchronization mechanism is en implicit synchronization mechanism.

50. The system of claim 49 wherein the implicit synchronization mechanism is a reliable network transport protocol.

51. The system of claim 47 wherein the synchronization mechanism is a reliable network transport protocol.

52. The system of claim 39 wherein responsive to the data block being unencoded, the decoder module stores the contents of the respective received data block as a previously received data block further comprises determining whether to delete at least one of the previously received data blocks.

53. The system of claim 39 wherein the decoder module, responsive to no match in contents between the respective received data block and any one of the previously received data blocks, stores the respective received data block in the least recently used data structure, and associates the position of most recently used with the respective received data block.

54. The system of claim 53 further comprising:

responsive to the least recently used data structure being at the maximum capacity, storing the respective received data block in the least recently used data structure comprises replacing the previously received data block having the order position of least recently used data block with the respective received data block.

58. The system of claim 39 wherein the system further comprises a decapsulation module for decapsulating a block of data received over the network from a source address supported by a corresponding encoder module.

59. The system of claim 58 wherein the received data block is included in a packet, and the packet has been decapsulated as one packet.

60. The system of claim 58 wherein the received data block is included in a packet and the packet has been decapsulated with at least one other packet in an outgoing packet transmitted from a source address supported by a corresponding encoder module.

61. The system of claim 58 wherein the decapsulation module decapsulates the received block of data at a first layer of a model describing the flow of data across a network and the decoder module decodes the received block of data at a second layer of the model.

62. The system of claim 61 wherein the first layer and the second layer are the same layer.

63. The system of claim 61 wherein one of the layers is a connection-oriented layer and the other layer is a connectionless layer.

55. A system for decoding one or more repetitive data blocks in data communicated over a network comprising:

a decoder module receiving data blocks from an encoder module;

a memory having a least recently used data structure for storing one or more previously received unique data blocks, each of the previously received unique data blocks having a unique identifier and a position from a most recently used to least recently used of the one or more previously received blocks;

a synchronization mechanism including a same size for the least recently used data structure as the size of a second least recently used data structure accessible by the corresponding encoder module that transmitted the respective received data block, a reliable network transport protocol being used for the transmission of the respective received data block,

wherein the decoder module determines whether the contents of each of the received data blocks is in encoded form,

wherein responsive to the respective data block being in encoded four, the decoder module selects the contents of a matching previously received block as the contents for the respective encoded block, and

wherein responsive to the data block being unencoded, the decoder module stores the contents of the respective received data block as a previously received data block.

56. A system for decoding one or more repetitive data blocks in data communicated over a network comprising:

a decoder module receiving data blocks from an encoder module;

a memory having a least recently used data structure for storing one or more previously received unique data blocks, each of the previously received unique data blocks having a unique identifier and a position from a most recently used to least recently used of the one or more previously received blocks;

a synchronization mechanism including a same size for the least recently used data structure as the size of a second least recently used data structure accessible by the corresponding encoder module, an indicator for each received data block indicating whether the received data block has been previously associated with the second least recently used data structure, and an installation acknowledgement transmitting by the decoder module to the corresponding encoder module responsive to an installation of each received block in the least recently used data structure,

wherein the decoder module determines whether the contents of each of the received data blocks is in encoded form,

wherein responsive to the respective data block being in encoded form, the decoder module selects the contents of a matching previously received block as the contents for the respective encoded block, and

wherein responsive to the data block being unencoded, the decoder module stores the contents of the respective received data block as a previously received data block.

57. A system for decoding one or more repetitive data blocks in data communicated over a network comprising:

a decoder module receiving data blocks from an encoder module;

a memory having a least recently used data structure for storing one or more previously received unique data blocks, each of the previously received unique data blocks having a unique identifier and a position from a most recently used to least recently used of the one or more previously received blocks;

a synchronization mechanism wherein the decoder module receives a version number of the previously received data block having the matching contents, the version number indicating bow many times the identifier for this data block has been re-used,

wherein the decoder module determines whether the contents of each of the received data blocks is in encoded form,

wherein responsive to the respective data block being in encoded form, the decoder module selects the contents of a matching previously received block as the contents for the respective encoded block, and

wherein responsive to the data block being unencoded, the decoder module stores the contents of the respective received data block as a previously received data block.

64. A system for decoding one or more repetitive data blocks in data communicated over a network comprising:

a decoder module that transmits to the encoder module over the network routing information in accordance with a routing protocol, the routing information including network topology information for the network for identifying one or more addresses of client devices that the decoder module supports to control selective encoding of data at the encoder module based on the client devices supported by the decoder module by determining with the encoder module whether client devices are supported for decoding by at least one corresponding decoder module based on destination addresses of the client devices and passing through the data when the destination addresses of the client devices are not supported for decoding by a decoder module, the decoder module receiving data blocks from an encoder module; and

a memory accessible to the decoder module for storing the contents of one or more of the data blocks previously received from the encoder module,

wherein the decoder module determines whether the contents of each of the received data blocks is in encoded form,

wherein responsive to the respective data block being in encoded form, the decoder module selects the contents of a matching previously received block as the contents for the respective encoded block, and

wherein responsive to the data block being unencoded, the decoder module stores the contents of the respective received data block as a previously received data block.

65. The system of claim 64 wherein the decoder module participates in one or more routing protocols for obtaining routing information.

66. A system for decreasing one or more repetitive data blocks in data communicated over a network comprising:

an encoder module, coupled in the network, the encoder module intercepting the data, the encoder module extracting data blocks from different communication sessions from the intercepted data, each extracted block having a destination address of a client device to which the extracted date block is destined, the encoder module determining whether the client device is supported for decoding by at least one corresponding decoder module based on the destination address of the client device, and the encoder module passing through the data when the destination address of the client device that is not supported for decoding by at least one decoder module;

a first memory, accessible to the encoder module, for storing the contents of one or more data blocks previously transmitted by the encoder module,

wherein the encoder module determines whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block,

wherein responsive to a match, the encoder module encodes the respective extracted data block and transmits the respective extracted data block in encoded form to the at least one corresponding decoder module, and responsive to no match, the encoder module transmits the respective extracted data block in intercepted form to the at least one corresponding decoder module;

the at least one corresponding decoder module being coupled in the network, the decoder module receiving data blocks for different communication sessions from the encoder module; and

a second memory accessible to the decoder module for storing the contents of one or more data blacks previously received from the encoder module,

wherein the at least one corresponding decoder module determines whether the contents of each of the received data blocks is in encoded form,

wherein responsive to the respective data block being in encoded form, the decoder module selects the contents of a matching previously received block as the contents for the respective encoded block, and responsive to the data block being unencoded, the decoder module stores the contents of the respective received data block as a previously received data block, and

wherein the decoder module transmits routing information in accordance with a routing protocol, the routing information including network topology information for the network for identifying one or more addresses it supports over the network to the encoder module.

67. A method for encoding one or more repetitive data blocks in data communicated over a network comprising:

receiving routing information in accordance with a routing protocol over the network from one or more decoder modules, the routing information including network topology information for the network;

determining, from the routing information, one or more destination addresses of client devices that is supported by each respective decoder module;

intercepting data communicated from a server to a client device;

extracting data blocks from different communication sessions from the intercepted data, each extracted block having a destination addresses for the client device to which the intercepted data in destined;

passing through the data when the destination address of the client device to which the intercepted data is destined is not one of the client devices supported for decoding by the one or more decoder modules;

storing the contents of one or more data blocks previously transmitted;

determining whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block;

responsive to a match encoding the respective extracted data block, and transmitting the respective extracted data block in encoded form to the at least one corresponding decoder module; and

responsive to no match, the encoder module transmitting over the network the respective extracted data block in intercepted form.

68. The method of claim 67 further comprising responsive to a match in contents, transmitting an indicator for identifying that the contents of a respective data block have been previously transmitted.

69. The method of claim 67 further comprising synchronizing the identification of the one or more previously transmitted data blacks with at least one corresponding decoder module supporting the destination address of the extracted data block.

70. The method of claim 67 further comprising, responsive to no match between the contents of the respective extracted data black and the contents of at least one of the previously transmitted blocks, determining whether to delete at least one of the previously transmitted data blocks in the memory, and storing the respective extracted data block in intercepted form in the memory.

71. The method of claim 67 wherein storing the contents of one or more data blocks previously transmitted comprises storing one or more previously transmitted unique data blocks in a least recently used data structure, said least recently used data structure having a maximum capacity and each of the previously transmitted unique data blocks having a unique identifier and a position in an order of most recently used to least recently used of the one or more stored blocks.

73. The method of claim 71 further comprising, responsive to no match between the contents of an extracted data block and the contents of one of the previously transmitted data blocks, storing the extracted data block in the least recently used data structure, and associating the position of most recently used with the extracted data block.

74. The method of claim 73 further comprising:

responsive to the least recently used data structure being at the maximum capacity,

deleting the previously transmitted data block having the order position of least recently used data block.

75. The method of claim 74 further comprising encoding at least one data block at a first layer of a model describing the flow of data across a network and the encapsulating the at least one extracted data block at a second layer of the model.

76. The method of claim 75 wherein one of the layers is a connection-oriented layer and the other layer is a connectionless layer.

77. The method of claim 67 further comprising determining network topology information based upon the routing information from the one or more decoder modules.

78. The method of claim 77 further comprising responsive to multiple decoder modules in the network supporting the same address, determining a destination decoder module for one or more extracted data blocks based upon network topology information and routing criteria.

72. A method for encoding one or more repetitive data blocks in data communicated over a network comprising:

intercepting data communicated from at least one server to at least one client device;

extracting data blocks from different communication sessions from the intercepted data, each extracted block having a destination address of the client device to which the extracted data block is destined and determining, based on the destination address of the client device, whether the client device is supported for decoding by at least one corresponding decoder module;

passing through the data when the destination address of the client device to which the intercepted data is destined is not supported for decoding by the at least one corresponding decoder module;

storing one or more previously transmitted unique data blocks in a least recently used data structure, each of the previously transmitted unique data blocks having a unique identifier and a position in an order of most recently used to least recently used of the one or more stored blocks;

determining whether there is a match between the contents of each of the extracted data blocks and the contents of at least one previously transmitted data block;

responsive to a match in contents, encoding the respective extracted data block, transmitting the respective extracted data block in encoded form to the at least one corresponding decoder module, and associating the previously transmitted data block having the matching contents with the position in the least recently used data structure indicating the most recently used previously transmitted data block; and

responsive to no match, transmitting over the network the respective extracted data block in intercepted form.

79. A method for decoding one or more repetitive data blocks in data communicated over a network comprising:

transmitting routing information in accordance with a routing protocol for identifying one or more addresses of client devices that a decoder module supports over a network to an encoder module, to control selective encoding of data at the encoder module based on the client devices supported by the decoder module by determining with the encoder module whether client devices are supported for decoding by at least one corresponding decoder module based on destination addresses of the client devices and passing through the data when the destination addresses of the client devices are not supported for decoding by a decoder module, wherein the routing information includes network topology information for the network;

receiving data blocks with the decoder module for different communication sessions from the encoder module;

storing the contents of one or more data blocks previously received from the encoder module;

determining whether the contents of each of the received data blocks is in encoded form;

responsive to the respective data block being in encoded form, selecting the contents of a matching previously received block as the contents for the respective encoded block; and

responsive to the data block being unencoded, storing the contents of the respective received data block as a previously received data block.

80. The method of claim 79 further comprising receiving an indicator for identifying that the contents of at least one encoded data block have been previously transmitted to the decoder module.

81. The method of claim 79 further comprising the identification of the one or more previously received data blocks from a corresponding encoder module.

82. The method of claim 79 further comprising responsive to the data block being unencoded, storing the contents of the respective received data block as a previously received data block further comprises determining whether to delete at least one of the previously received data blocks.

83. The method of claim 79 wherein storing the previously received data blocks further comprises storing one or more previously received unique data blocks in a least recently used data structure, the least recently used data structure having a maximum capacity and each of the previously received unique data blocks having a unique identifier and a position in an order of most recently used to least recently used of the one or more previously received blocks.

84. The method of claim 83 further comprising responsive to a match in contents, associates the previously received data block having the matching contents with the position in the least recently used data structure indicating the most recently used previously received data block.

85. The method of claim 83 further comprising responsive to no match in contents between the respective received data block and any one of the previously received data blocks, storing the respective received data block in the least recently used data structure, and associating the position of most recently used with the respective received data block.

86. The method of claim 85 further comprising:

responsive to the least recently used data structure being at the maximum capacity, deleting the previously received data block having the order position of least recently used data block.

87. The method of claim 79 further comprising decapsulating the received block of data at a first layer of a model describing the flow of data across a network and decoding the received block of data at a second layer of the model.

88. The method of claim 87 wherein the first layer and the second layer are the same layer.

89. The method of claim 88 wherein one of the layers is a connection-oriented layer and the other layer is a connectionless layer.