MATCH ENGINE FOR DETECTION OF MULTI-PATTERN RULES

• US 20140052748A1
• Filed: 10/25/2013
• Published: 02/20/2014
• Est. Priority Date: 07/31/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: van Lunteren, Jan
• Granted Patent: US 9,015,102 B2
• Time in Patent Office: Days
• US Class Current: 707/758
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 21/561 Virus type analysis
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

Methods, systems and computer program products are disclosed for detecting patterns in a data stream that match multi-pattern rules. One embodiment of the invention provides a method of recognizing a specified group of patterns in a data stream. The method comprises identifying a rule for said specified group of patterns in the data stream, and using a first array of finite state machines to scan the data stream for at least some of the patterns in the specified group. For patterns in the specified group that are found in the data stream by the first array of finite state machines, pattern identifiers are sent to a second array of finite state machines. The second array of finite state machines determines if the specified group of patterns is in the data stream in accordance with the identified rule by, at least in part, using said pattern identifiers.

Claims

1. A method of recognizing a specified group of patterns in a data stream, the method comprising:

identifying a multi-pattern rule for said specified group of patterns in the data stream;

using a first array of finite state machines, in a first, pattern scanner stage, to scan the data stream for at least some of the patterns in the specified group;

for patterns in the specified group that are found in the data stream by the first array of finite state machines, sending pattern identifiers to a second array of finite state machines, in a second, rule processor stage;

using the second array of finite state machines for determining if the specified group of patterns is in the data stream in accordance with the identified multi-pattern rule by, at least in part, using said pattern identifiers; and

using a compiler to distribute a function that matches an individual pattern over the first, pattern scanner stage and the second, rule processor stage.

2. The method according to claim 1, wherein the method is used to recognize a multitude of specified groups of patterns in the data stream, and wherein the second array of finite state machines determining if the specified group of patterns is in the data stream includes:

running a multitude of threads on the second array of finite state machines; and

using said multitude of threads to determine whether the multitude of specified groups of patterns are in the data stream.

3. The method according to claim 2, wherein the using said multitude of threads includes using each of the threads to determine whether a respective one of the specified groups of patterns is present in the data stream.

4. The method according to claim 2, wherein the identifying a rule includes identifying a respective one rule for each of the multitude of specified groups of patterns.

5. The method according to claim 4, wherein the using the first array includes using the first array to scan the data stream for at least some of the patterns in each of said multitude of specified groups of patterns.

6. The method according to claim 2, wherein the sending pattern identifiers to the second array includes:

for each of the multitude of specified groups of patterns, for patterns in said each group that are found in the data stream by the first array of finite state machines, sending pattern identifiers to an associated one of the threads running on the second array of finite state machines.

7. The method according to claim 2, wherein:

each of the threads is used to determine whether a respective one of the specified groups of patterns is in the data stream; and

the sending pattern identifiers to the second array includes, for each of the specified groups of patterns, for patterns in said each group that are found in the data stream by the first array of finite state machines, sending pattern identifiers to the one of the threads used to determine whether said each of the specified groups of patterns is in the data stream.

8. The method according to claim 6, wherein the sending pattern identifiers further includes using an ID distributor to ensure that the only pattern identifiers received by each thread are the pattern identifiers needed by said each thread to determine whether the associated one of the specified groups of patterns is in the data stream.

9. The method according to claim 6, wherein each thread has a respective one thread ID, and wherein the sending pattern identifiers includes, in each pattern identifier, the thread ID of the thread to which said pattern identifier is sent.

10. The method according to claim 1, wherein the using the first array of finite state machines includes using the first array of finite state machines to scan for all of the patterns in the specified group.

11. A finite state machine engine for determining whether a multitude of specified groups of patterns are in a data stream, the machine comprising:

a context memory holding a multitude of thread context records;

a transition rule memory holding a multitude of multi-pattern rules for transitioning between different states of the machine;

a rule selector for selecting one or more of the multi-pattern rules from the transition rule memory;

an array of finite state machines for receiving, in a first, pattern scanner stage, input data identifying patterns, of the specified groups of patterns, that are in the data stream, and for processing said input data and data from the context memory, in a second rule processor stage, to determine whether the multitude of specified groups of patterns are in the data stream in accordance with specified multi-pattern rules selected from the transition rule memory; and

a compiler to distribute a function that matches an individual pattern over the first, pattern scanner stage and the second, rule processor stage.

12. The finite state machine engine according to claim 11, wherein the context records hold state registers, table registers, and mask registers.

13. The finite state machine engine according to claim 12, wherein each of the context records is used to determine whether a respective one of the multitude of specified groups of patterns is in the data stream in accordance with an associated one of the specified rules.

14. The finite state machine engine according to claim 13, wherein the input data includes pattern IDs, and the context records are selected on the basis of information contained in the pattern IDs.

15. The finite state machine engine according to claim 11, wherein each context record is related to one or multiple multi-pattern rules.

16. An article of manufacture comprising:

at least one computer usable medium having computer readable program code logic to execute a machine instruction in a processing unit for using a computer to for recognizing a specified group of patterns in a data stream in accordance with a defined multi-pattern rule, said computer readable program code logic, when executing, performing the following;

using a first array of finite state machines to scan the data stream for at least some of the patterns in the specified group;

for patterns in the specified group that are found in the data stream by the first array of finite state machines, sending pattern identifiers to a second array of finite state machines; and

using the second array of finite state machines for determining if the specified group of patterns is in the data stream in accordance with the identified multi-pattern rule by, at least in part, using said pattern identifiers.

17. The article of manufacture according to claim 16, wherein the machine instruction is executed to use the computer for recognizing a multitude of specified groups of patterns in the data stream, and wherein the second array of finite state machines determining if the specified group of patterns is in the data stream includes:

running a multitude of threads on the second array of finite state machines; and

using said multitude of threads to determine whether the multitude of specified groups of patterns are in the data stream.

18. The article of manufacture according to claim 17, wherein:

each of the threads is used to determine whether a respective one of the specified groups of patterns is in the data stream; and

the sending pattern identifiers to the second array includes, for each of the specified groups of patterns, for patterns in said each group that are found in the data stream by the first array of finite state machines, sending pattern identifiers to the one of the threads used to determine whether said each of the specified groups of patterns is in the data stream.

19. The article of manufacture according to claim 18, wherein the sending pattern identifiers further includes using an ID distributor to ensure that the only pattern identifiers received by each thread are the pattern identifiers needed by said each thread to determine whether the associated one of the specified groups of patterns is in the data stream.

20. The article of manufacture according to claim 19, wherein each thread has a respective one thread ID, and wherein the sending pattern identifiers including, in each pattern identifier, the thread ID of the thread to which said pattern identifier is sent.