System and method for data feed acceleration and encryption

• US 7,417,568 B2
• Filed: 05/07/2003
• Issued: 08/26/2008
• Est. Priority Date: 10/03/2000
• Status: Expired due to Fees

• Current Assignee: Realtime Data LLC (Realtime Data Compression Systems, Inc.)
• Original Assignee: Realtime Data LLC (Realtime Data Compression Systems, Inc.)
• Inventors: Fallon, James J., Pickel, Paul F., McErlain, Stephen J., Melone, Carlton J.
• Primary Examiner(s): JEANGLAUDE, JEAN BRUNER
• Application Number:
  US10/434,305
• Publication Number:
  US 20040042506A1
• Time in Patent Office: 1,938 Days
• Field of Search: 341/51, 341/50, 370/355, 380/212, 709/238, 358/400, 345/419, 714/752
• US Class Current: 341/51
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G06Q 50/188 Electronic negotiation
• CPC Class Codes: H04L 12/1895 for short real-time information, e.g. alarms, notifications, alerts, updates
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/55 Push-based network services
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

Systems and methods for providing accelerated transmission of broadcast data, such as financial data and news feeds, over a communication channel using data compression and decompression to provide secure transmission and transparent multiplication of communication bandwidth, as well as reduce the latency associated with data transmission of conventional systems.

Claims

1. A method for compressing data, wherein one or more types of encoding are applied to a data stream depending on identifiable data fields in the data stream, the method comprising:

recognizing a data field type of a data field in the data stream, wherein the data field is included in a packet;

selecting an encoder associated with the recognized data field type;

encoding the data in the data field with the selected encoder; and

providing a descriptor with the encoded data which identifies the selected encoder.

2. The method of claim 1, wherein selecting an encoder associated with the recognized data field type is further based on a packet type of the packet.

3. The method of claim 2, further comprising using a packet independent encoder for encoding a UDP (User Datagram Protocol) data packet.

4. The method of claim 2, further comprising selecting packet independent or dependent encoders based on a packet type.

7. The method of claim 1, further comprising using arithmetic encoding with single character codes to encode data of single character fields.

8. The method of claim 1, further comprising using Huffman encoding with small set multiple character codes to encode data of multiple character fields.

9. The method of claim 1, further comprising using variable length encoding with large set multiple character codes to encode data of multiple character fields.

10. The method of claim 9, wherein large set multiple character codes utilize truncation of trailing spaces.

11. The method of claim 1, further comprising using table-based encoding with large set multiple character codes to encode data of multiple character fields.

12. The method of claim 1, further comprising using variable length encoding with variable length codes to encode absolute number values.

13. The method of claim 1, further comprising using numerical difference encoding to encode relative numeric values.

14. The method of claim 1, further comprising encoding time data based on a difference between a previous time data.

15. The method of claim 1, further comprising difference encoding based on a first absolute values within a same data packet for packet independent data compression.

16. The method of claim 1, comprising the step of using difference encoding to encode a message sequence number.

17. The method of claim 1, wherein the packet comprises messages, a date is encoded only one time in the packet if all messages in the packet have the same date.

18. The method of claim 17, further comprising including date data as part of a message of the packet only if the message has a different date than the date.

19. The method of claim 1, further comprising including error detection codes in a compressed data packet.

25. The method of claim 1, wherein the encoding is performed in a server.

26. The method of claim 1, wherein the data stream comprises financial data.

27. The method of claim 1, wherein the data stream comprises news data.

28. The method of claim 1, further comprising transmitting the encoded data and the descriptor over a communications channel to a computer.

29. The method of claim 1, further comprising transmitting the encoded data and the descriptor over a communications channel to a computer, wherein the encoding is performed on a server separate from the computer.

30. The method of claim 1, further comprising transmitting the encoded data and the descriptor over a communications channel to a computer, wherein the encoding is performed on a server separate from the computer and the encoding of the data and the transmitting of the encoded data occurs faster than the data is able to be transmitted in an uuencoded form.

31. The method of claim 1, wherein the encoding occurs in real time.

32. The method of claim 1, wherein the encoding does not require packet-to-packet data dependency.

33. The method of claim 1, wherein the encoded data is broadcasted over a communications channel.

34. The method of claim 1, wherein the selected encoder is a Huffman encoder.

35. The method of claim 1, wherein the selected encoder is a Lempel Ziv encoder.

36. The method of claim 1, wherein the selected encoder is an Arithmetic encoder.

37. The method of claim 1, wherein the encoded data is provided through a landline and then to and through a satellite system.

38. The method of claim 1, wherein the encoded data is provided to a landline and then to and through a modem.

39. The method of claim 1, wherein the packet is a UDP packet.

40. The method of claim 1, further comprising compressing data of an unrecognized data field using one encoder.

41. The method of claim 1, further comprising compressing data of an unrecognized field using a plurality of encoders.

45. The method of claim 1, further comprising compressing data of an unrecognized field using a plurality of encoders sequentially.

46. The method of claim 1, wherein the encoded data comprises financial data and the encoded data is representative of a compression ratio greater than 10:

1.

47. The method of claim 1, wherein the encoded data is representative of a compression ratio greater than 10:

1.

48. The method of claim 1, wherein the encoded data is representative of a compression ratio greater than 10:

1 and the encoded data is transmitted over a communications channel.

49. The method of claim 1, wherein the encoded data includes an inside bid.

50. The method of claim 1, wherein the latency associated with the encoding is under 10 milliseconds.

51. The method of claim 1, wherein the latency associated with the encoding is under 128 milliseconds.

5. A method for compressing data, wherein one or more types of encoding are applied to a data stream depending on identifiable data fields in the data stream, the method comprising:

recognizing a data field type of a data field in a data stream;

selecting an encoder associated with the recognized data field type;

encoding the data in the data field with the selected encoder;

providing a descriptor with the encoded data which identifies the selected encoder; and

compressing data of an unrecognized data field using content independent data compression.

6. A method for compressing data, wherein one or more types of encoding are applied to a data stream depending on identifiable data fields in the data stream, the method comprising:

recognizing a data field type of a data field in a data stream;

selecting an encoder associated with the recognized data field type;

encoding the data in the data field with the selected encoder;

providing a descriptor with the encoded data which identifies the selected encoder; and

compressing data of an unrecognized packet type using content independent data compression.

20. A method for creating a data feed dependent data compression routine, the method comprising:

creating a description file that describes one or more data fields and one or more encoders associated with each data field;

processing the description file with a data compression compiler; and

outputting an executable file that is used to process a stream of data by recognizing data field types in the data stream and applying encoders associated with the recognized data field types to encode the data stream.

21. The method of claim 20, wherein the output file comprises a dynamic link library.

22. The method of claim 20, wherein the executable file comprises instructions for performing data compression on unrecognized data fields.

23. The method of claim 22, wherein encoder types that are used for the data compression on unrecognized data fields are specified in the description file.

24. A method for creating a data feed dependent data compression routine, the method comprising:

creating a description file that describes one or more data fields and one or more encoders associated with each data field;

processing the description file with a data compression compiler; and

outputting source code that is used to process a stream of data by recognizing data field types in the data stream and applying encoders associated with the recognized data field types to encode the data stream.

42. A method for compressing data, wherein one or more types of encoding are applied to a data stream depending on identifiable data fields in the data stream, the method comprising:

recognizing a data field type of a data field in the data stream, wherein the data field is included in a packet;

selecting an encoder associated with the recognized data field type;

encoding the data in the data field with the selected encoder;

providing a descriptor with the encoded data which identifies the selected encoder; and

compressing data of an unrecognized field using a plurality of encoders and determining the compression ratio associated with each one of the plurality of encoders.

43. A method for compressing data, wherein one or more types of encoding are applied to a data stream depending on identifiable data fields in the data stream, the method comprising:

recognizing a data field type of a data field in the data stream, wherein the data field is included in a packet;

selecting an encoder associated with the recognized data field type;

encoding the data in the data field with the selected encoder;

providing a descriptor with the encoded data which identifies the selected encoder; and

compressing data of an unrecognized field using a plurality of encoders and determining the compression ratio associated with each one of the plurality of encoders, wherein a compressed data block is provided based on the determination.

44. A method for compressing data, wherein one or more types of encoding are applied to a data stream depending on identifiable data fields in the data stream, the method comprising:

recognizing a data field type of a data field in the data stream, wherein the data field is included in a packet;

selecting an encoder associated with the recognized data field type;

encoding the data in the data field with the selected encoder;

providing a descriptor with the encoded data which identifies the selected encoder; and

compressing data of an unrecognized field using a plurality of encoders in parallel.

52. A method for creating a data feed dependent data compression routine, the method comprising:

creating a description file that describes one or more data fields and one or more encoders associated with each data field;

processing the description file; and

outputting code that is used to process a stream of data by recognizing data field types in the data stream and applying encoders associated with the recognized data field types to encode the data stream.

53. A method comprising:

recognizing a data field type of a data field in a financial data feed;

selecting an encoder associated with the recognized data field type;

compressing the data in the data field with the selected encoder to provide compressed data;

broadcasting the compressed data to a plurality of systems, wherein the compressing achieves a compression ratio of over 10;

1; and

utilizing a descriptor, which identifies the selected encoder, to decompress the compressed data on at least one of the plurality of systems.

54. The method of claim 53, wherein the compressing is performed in a server.

55. The method of claim 53, wherein the compressing occurs in real time.

56. The method of claim 53, wherein the compressing does not require packet-to-packet data dependency.

57. The method of claim 53, wherein the selected encoder is a Huffman encoder.

58. The method of claim 53, wherein the selected encoder is a Lempel Ziv encoder.

59. The method of claim 53, wherein the selected encoder is an Arithmetic encoder.

60. The method of claim 53, wherein the encoded data is provided to a landline and then to and through a modem.

61. The method of claim 53, wherein the data field is included in a packet and the packet is a UDP packet.

62. The method of claim 53, further comprising compressing data of an unrecognized data field using one encoder.

63. The method of claim 53, further comprising compressing data of an unrecognized field using a plurality of encoders.

68. The method of claim 53, wherein the compressed data includes an inside bid.

69. The method of claim 53, wherein the latency associated with the compressing is under 10 milliseconds.

70. The method of claim 53, wherein the latency associated with the compressing is under 128 milliseconds.

64. A method comprising:

recognizing a data field type of a data field in a financial data feed;

selecting an encoder associated with the recognized data field type;

compressing the data in the data field with the selected encoder to provide compressed data;

broadcasting the compressed data to a plurality of systems, wherein the compressing achieves a compression ratio of over 10;

1;

utilizing a descriptor, which identifies the selected encoder, to decompress the compressed data on at least one of the plurality of systems; and

compressing data of an unrecognized field using a plurality of encoders and determining the compression ratio associated with each one of the plurality of encoders.

65. A method comprising:

recognizing a data field type of a data field in a financial data feed;

selecting an encoder associated with the recognized data field type;

compressing the data in the data field with the selected encoder to provide compressed data;

broadcasting the compressed data to a plurality of systems, wherein the compressing achieves a compression ratio of over 10;

1;

utilizing a descriptor, which identifies the selected encoder, to decompress the compressed data on at least one of the plurality of systems; and

compressing data of an unrecognized field using a plurality of encoders and determining the compression ratio associated with each one of the plurality of encoders, wherein a compressed data block is provided based on the determination.

66. A method comprising:

recognizing a data field type of a data field in a financial data feed;

selecting an encoder associated with the recognized data field type;

compressing the data in the data field with the selected encoder to provide compressed data;

broadcasting the compressed data to a plurality of systems, wherein the compressing achieves a compression ratio of over 10;

1;

utilizing a descriptor, which identifies the selected encoder, to decompress the compressed data on at least one of the plurality of systems; and

compressing data of an unrecognized field using a plurality of encoders in parallel.

67. A method comprising:

recognizing a data field type of a data field in a financial data feed;

selecting an encoder associated with the recognized data field type;

compressing the data in the data field with the selected encoder to provide compressed data;

broadcasting the compressed data to a plurality of systems, wherein the compressing achieves a compression ratio of over 10;

1;

utilizing a descriptor, which identifies the selected encoder, to decompress the cornpressed data on at least one of the plurality of systems; and

compressing data of an unrecognized field using a plurality of encoders sequentially.