Systems and methods for providing real-time classification of continuous data streams

• US 7,937,269 B2
• Filed: 08/22/2005
• Issued: 05/03/2011
• Est. Priority Date: 08/22/2005
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Yu, Philip Shilung, Aggarwal, Charu Chandra
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): Adesanya; Olujimi A
• Application Number:
  US11/208,893
• Publication Number:
  US 20070043565A1
• Time in Patent Office: 2,080 Days
• Field of Search: 704/9, 704/214, 704/238, 704/245, 704/256, 704/272
• US Class Current: 704/245
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques

Abstract

Systems and methods are provided for real-time classification of streaming data. In particular, systems and methods for real-time classification of continuous data streams implement micro-clustering methods for offline and online processing of training data to build and dynamically update training models that are used for classification, as well as incrementally clustering the data over contiguous segments of a continuous data stream (in real-time) into a plurality of micro-clusters from which target profiles are constructed which define/model the behavior of the data in individual segments of the data stream.

Claims

1. A method for real-time classification of a continuous data stream by a data stream processing server comprising a processor and implementing a classification module, the method comprising:

receiving a continuous data stream by the data stream processing server;

clustering, incrementally, a set of data records in each contiguous segment of the received data stream into a plurality of micro-clusters, wherein the plurality of micro-clusters is stored as a snapshot in time, the snapshot updating with time and indicating a dominant micro-cluster in the data stream;

generating a target profile for each segment of the received data stream based on the snapshot of micro-clusters associated with each segment, wherein generating the target profile comprises generating a histogram profile for a given segment using summary information of data records associated with the micro-clusters for the given segment, wherein the histogram profile is generated based on relative frequencies of data points associated with each micro-cluster for the given segment as compared to a total number of data points in the micro-clusters for the given segment; and

classifying, by the classification module, each segment of the received data stream using the target profile associated with each segment.

2. The method of claim 1, wherein classifying comprises classifying each segment by matching the target profile for a given segment to a similar training profile.

3. The method of claim 2, wherein matching comprises determining a distance between the target profile and each of a plurality of training profiles to determine a training profile that is closest to the target profile.

4. The method of claim 3, wherein determining a distance between the target profile and a training profile comprises determining a Manhattan distance between said profiles.

5. The method of claim 3, wherein determining a distance between the target profile and a training profile comprises determining a Euclidean distance between said profiles.

6. The method of claim 1, wherein classifying comprises comparing the target profiles over a plurality of segments to detect a data pattern in the data stream.

7. The method of claim 1, wherein classifying comprises comparing the target profiles over a plurality of segments to detect an event in the data stream.

8. The method of claim 1, wherein classifying comprises comparing the target profiles over a plurality of segments to cluster the segments into groups of segments.

9. The method of claim 1, wherein the continuous data stream is a voice data stream.

10. The method of claim 9, wherein classifying comprises performing real-time speaker identification of the voice data stream.

11. The method of claim 9, wherein the voice data stream comprises a stream of VoIP packets.

12. A non-transitory program storage device readable by machine, embodying a program of instructions executable by a processor of the machine to perform method steps for real-time classification of a continuous data stream, the method steps comprising:

receiving a continuous data stream;

clustering, incrementally, speech data in each contiguous segment of the received data stream into a plurality of micro-clusters, wherein the plurality of micro-clusters is stored as a snapshot in time, the snapshot updating with time and indicating a dominant micro-cluster in the data stream;

generating a target profile for each segment of the received data stream based on the snapshot of micro-clusters associated with each segment, wherein generating the target profile comprises generating a histogram profile for a given segment using summary information of data records associated with the micro-clusters for the given segment, wherein the histogram profile is generated based on relative frequencies of data points associated with each micro-cluster for the given segment as compared to a total number of data points in the micro-clusters for the given segment; and

classifying each segment of the received data stream using the target profile associated with each segment.

13. The non-transitory program storage device of claim 12, wherein the instructions for classifying comprise instructions for classifying each segment by matching the target profile for a given segment to a similar training profile.

14. The non-transitory program storage device of claim 13, wherein the instructions for matching comprise instructions for determining a distance between the target profile and each of a plurality of training profiles to determine a training profile that is closest to the target profile.

15. The non-transitory program storage device of claim 14, wherein the instructions for determining a distance between the target profile and a training profile comprise instructions for determining a Manhattan distance between said profiles.

16. The non-transitory program storage device of claim 14, wherein the instructions for determining a distance between the target profile and a training profile comprise instructions for determining a Euclidean distance between said profiles.

17. The non-transitory program storage device of claim 12, wherein the instructions for classifying comprise instructions for comparing the target profiles over a plurality of segments to detect a data pattern in the data stream.

18. The non-transitory program storage device of claim 12, wherein the instructions for classifying comprise instructions for comparing the target profiles over a plurality of segments to detect an event in the data stream.

19. The non-transitory program storage device of claim 12, wherein the instructions for classifying comprise instructions for comparing the target profiles over a plurality of segments to cluster the segments into groups of segments.

20. The non-transitory program storage device of claim 12, wherein the continuous data stream is a voice data stream.

21. The non-transitory program storage device of claim 20, wherein the instructions for classifying comprise instructions for performing real-time speaker identification of the voice data stream.

22. The non-transitory program storage device of claim 20, wherein the voice data stream comprises a stream of VoIP packets.

23. A system for processing continuous data streams, comprising:

a data stream capturing system to capture a target data stream over a network;

a data clustering system that incrementally clusters a set of data records in each contiguous segment of the captured data stream into a plurality of micro-clusters, wherein the plurality of micro-clusters is stored as a snapshot in time, the snapshot updating with time and indicating a dominant micro-cluster in the target data stream, and generates a target profile for each segment of the captured data stream based on the snapshot of micro-clusters associated with each segment, wherein generating the target profile comprises generating a histogram profile for a given segment using summary information of data records associated with the micro-clusters for the given segment, wherein the histogram profile is generated based on relative frequencies of data points associated with each micro-cluster for the given segment as compared to a total number of data points in the micro-clusters for the given segment; and

a classification system to classify each segment of the captured data stream using the target profile associated with each segment.