System and method for managing data streams

• US 7,631,074 B1
• Filed: 09/30/2005
• Issued: 12/08/2009
• Est. Priority Date: 06/07/2005
• Status: Active Grant

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Srivastava, Divesh, Zhang, Rui, Koudas, Nick
• Primary Examiner(s): Harrell; Robert B
• Application Number:
  US11/240,518
• Time in Patent Office: 1,530 Days
• Field of Search: 709/224
• US Class Current: 709/224
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

A system for a data stream management system includes a filter transport aggregate for a high speed input data stream with a plurality of packets each packet comprising attributes. The system includes an evaluation system to evaluate the high speed input data stream and partitions the packets into groups the attributes and a table, wherein the table stores the attributes of each packets using a hash function. A phantom query is used to define partitioned groups of packets using attributes other than those used to group the packets for solving user queries without performing the user queries on the high speed input data stream.

Claims

1. A datastream management system comprising:

a. a filter transform aggregate for managing a high speed input data stream with a plurality of packets, wherein each packet comprises at least one attribute, and wherein the filter transform aggregate comprises;

i. an evaluation system to evaluate the high speed input data stream and to partition the plurality of packets into groups using similar attributes;

ii. a table to store the attributes of the plurality of packets using a hash function; and

iii. computer instruction to form a phantom query, wherein the phantom query uses a partitioned group of packets with attributes different than those used to group the plurality of packets into groups for solving user queries without performing the user queries on the high speed input data stream;

b. a memory storing the filter transform aggregate; and

c. a processor operating the filter transform aggregate.

2. The system of claim 1, wherein the attributes are source IP addresses, source ports, destination ports, destination IP addresses, time lengths, or combinations thereof

3. The system of claim 1, wherein the evaluation system is an SQL evaluation system, an Oracle evaluation system, or combinations thereof.

4. The system of claim 1, wherein the filter transform aggregate further comprises a low level filter transport aggregate and a high level filter transform aggregate.

5. The system of claim 1, wherein the filter transform aggregate comprises from about 10,000 packets to about 1,000,000 packets.

6. The system of claim 1, wherein the phantom query comprises a member selected from the group consisting of a request for source port, a request for destination port, an IP destination address, IP origination address, and combinations thereof.

7. The system of claim 1, wherein the user queries are performed on an output from the phantom query.

8. A memory storing a set of instructions executable by a processor, the instructions operable to:

a. create a filter transform aggregate;

b. administer a data stream with a plurality of packets, using the filter transform aggregate, wherein each packet comprises a plurality of attributes;

c. perform an evaluation on the data stream and partition the plurality of packets into groups by based on similar attributes;

d. apply a hash function to create a table in memory based on the attributes of the plurality of packets; and

e. make a phantom query for a defined partitioned group of packets using attributes other than those used to group the packets, to simultaneously solve at least two user queries by applying a phantom query to the high speed data stream.

9. The memory of claim 8, wherein the step of creating the filter transform aggregate comprises creating a low level filter transform aggregate and a high level filter transform aggregate.

10. The memory of claim 8, wherein the step of creating the filter transform aggregate further comprises creating at least three levels of filter transform aggregates.

11. The memory of claim 8, wherein the filter aggregate comprises from about 10,000 packets to about 1,000,000 packets.

12. The memory of claim 8, wherein the at least two user queries are performed on an output from the phantom query.

13. The memory of claim 8, wherein the data stream is a high speed input data stream.

14. The memory of claim 8, wherein the evaluation is performed by an SQL evaluation system, an Oracle evaluation system, or combinations thereof.

15. The memory of claim 8, wherein the attributes are source IP addresses, source ports, destination ports, destination IP addresses, time lengths, or combinations thereof.

16. The memory of claim 8, wherein the phantom query comprises a member selected from the group consisting of a request for source port, a request for destination port, an IP destination address, IP origination address, and combinations thereof.

17. A memory storing a set of instructions executable by a processor, the instructions operable to:

a. create a low level filter transform aggregate;

b. create a high level filter transform aggregate;

c. administer a high input data stream with a plurality of packets, wherein each packet comprises a plurality of attributes;

d. perform an evaluation on the high input data stream, and partition the plurality of packets into groups by an attribute;

e. create a table in memory based on the attributes of each packets using a hash function; and

f. make a phantom query for a partitioned group of packets using attributes different than those used to group the plurality of packets to solve simultaneously at least two user queries by applying the phantom query to the high input data stream.

18. The memory of claim 17, wherein the at least two user queries are performed on an output from the phantom query.

19. The memory of claim 17, wherein the attributes comprise a member selected for the group consisting of:

source IP addresses, source ports, destination ports, destination IP addresses, time lengths, and combinations thereof.

20. The memory of claim 17, wherein the phantom query comprises a member selected from the group consisting of a request for source port, a request for destination port, an IP destination address, IP origination address, and combinations thereof.