System and method for managing data streams

• US 8,117,307 B2
• Filed: 10/23/2009
• Issued: 02/14/2012
• Est. Priority Date: 06/07/2005
• Status: Expired due to Fees

• Current Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Srivastava, Divesh, Koudas, Nick, Zhang, Rui
• Primary Examiner(s): Harrell, Robert B
• Application Number:
  US12/605,033
• Publication Number:
  US 20100042606A1
• Time in Patent Office: 844 Days
• Field of Search: 709/224
• US Class Current: 709/224
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

A system for a data stream management system includes a filter transport aggregate for a high speed input data stream with a plurality of packets each packet comprising attributes. The system includes an evaluation system to evaluate the high speed input data stream and partitions the packets into groups the attributes and a table, wherein the table stores the attributes of each packets using a hash function. A phantom query is used to define partitioned groups of packets using attributes other than those used to group the packets for solving user queries without performing the user queries on the high speed input data stream.

Claims

1. A memory storing a set of instructions executable by a processor, the instructions operable to:

a. administer a data stream with a plurality of packets, using a filter transform aggregate, wherein each packet comprises a plurality of attributes;

b. partition the plurality of packets into groups based on similar attributes;

b. create a table in memory based on the attributes of the plurality of packets; and

c. make a phantom query for a defined partitioned group of packets using attributes other than those used to group the packets, to solve at least two user queries.

2. The memory of claim 1, wherein the step of creating the filter transform aggregate comprises creating a low level filter transform aggregate and a high level filter transform aggregate.

3. The memory of claim 1, wherein the step of creating the filter transform aggregate further comprises creating at least three levels of filter transform aggregates.

4. The memory of claim 1, wherein the filter aggregate comprises from about 10,000 packets to about 1,000,000 packets.

5. The memory of claim 1, wherein the at least two user queries are performed on an output from the phantom query.

6. The memory of claim 1, wherein the data stream is a high speed input data stream.

7. The memory of claim 1, wherein the evaluation is performed by an SQL evaluation system, an Oracle evaluation system, or combinations thereof.

8. The memory of claim 1, wherein the attributes are source IP addresses, source ports, destination ports, destination IP addresses, time lengths, or combinations thereof.

9. The memory of claim 1, wherein the phantom query comprises a member selected from the group consisting of a request for source port, a request for destination port, an IP destination address, IP origination address, and combinations thereof.

10. A datastream management system comprising:

a. a filter transform aggregate for managing a high speed input data stream with a plurality of packets, wherein each packet comprises at least one attribute, and wherein the filter transform aggregate comprises;

i. an evaluation system to partition the plurality of packets into groups using similar attributes;

ii. a table to store the attributes of the plurality of packets; and

iii. computer instruction to form a phantom query, wherein the phantom query uses a partitioned group of packets with attributes different than those used to group the plurality of packets into groups for solving user queries;

b. a memory storing the filter transform aggregate; and

c. a processor operating the filter transform aggregate.

11. The system of claim 10, wherein the attributes are source IP addresses, source ports, destination ports, destination IP addresses, time lengths, or combinations thereof.

12. The system of claim 10, wherein the evaluation system is an SQL evaluation system, an Oracle evaluation system, or combinations thereof.

13. The system of claim 10, wherein the filter transform aggregate further comprises a low level filter transport aggregate and a high level filter transform aggregate.

14. The system of claim 10, wherein the filter transform aggregate comprises from about 10,000 packets to about 1,000,000 packets.

15. The system of claim 10, wherein the phantom query comprises a member selected from the group consisting of a request for source port, a request for destination port, an IP destination address, IP origination address, and combinations thereof.

16. The system of claim 10, wherein the user queries are performed on an output from the phantom query.

17. A memory storing a set of instructions executable by a processor, the instructions operable to:

a. create a low level filter transform aggregate;

b. create a high level filter transform aggregate;

c. administer a high input data stream with a plurality of packets, wherein each packet comprises a plurality of attributes;

d. partition the plurality of packets into groups using similar attributes;

e. create a table in memory based on the attributes of each packets; and

d. make a phantom query for a partitioned group of packets using attributes different than those used to group the plurality of packets to solve simultaneously at least two user queries by applying the phantom query to the high input data stream.

18. The memory of claim 17, wherein the at least two user queries are performed on an output from the phantom query.

19. The memory of claim 17, wherein the attributes are source IP addresses, source ports, destination ports, destination IP addresses, time lengths, and combinations thereof.

20. The memory of claim 17, wherein the phantom query comprises a member selected from the group consisting of a request for source port, a request for destination port, an IP destination address, IP origination address, and combinations thereof.