Method of Extracting Sections of a Data Stream

• US 20090276427A1
• Filed: 07/17/2009
• Published: 11/05/2009
• Est. Priority Date: 01/18/2007
• Status: Abandoned Application

• Current Assignee: ROKE Manor Research Limited (Chemring Group PLC)
• Original Assignee: ROKE Manor Research Limited (Chemring Group PLC)
• Inventors: DUXBURY, Neil
• Application Number:
  US12/505,147
• Publication Number:
  US 20090276427A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/295 Named entity recognition

Abstract

A method of extracting sections of a data stream, the sections including a set of sequences. Each sequence is encoded separately and coupled together to define a section. The method involves determining a combination of at least two sequences of the set, comparing the combination of sequences with sequences in the data stream, and rejecting or accepting extraction of the section of the data stream based upon the result of the comparison. If the combination of sequences does not include a start and end marker for the section, a search for the start and end markers is carried out before the section is extracted.

Claims

1. A method of extracting sections of a data stream comprising end point identifiers, the sections comprising a set of sequences;

wherein each sequence is encoded separately and coupled together to define the section;

the method comprising;

determining a combination of at least two sequences of the set;

comparing the combination of sequences with sequences in the data stream; and

rejecting or accepting extraction of the section of the data stream based upon the result of the comparison,

wherein when the combination of sequences does not include a start and end marker for the section, a search for the start and end markers is carried out before the section is extracted.

2. A method according to claim 1, wherein extraction of the section is accepted when the combination of sequences in any order matches sequences in the section of the data stream.

3. A method according to claim 1, wherein extraction of the section is rejected when the combination of sequences does not match any of the sequences in the section of the data stream, and

wherein the search continues for further instances of the combination of sequences.

4. A method according to claim 1, wherein a sequence comprises a series of bits having a predetermined format, including an anchor or a bridge.

5. A method according to claim 4, wherein the anchor is a statistically rare, or low probability sequence in the data stream.

6. A method according to claim 5, wherein the probability of occurrence is less than 1%.

7. A method according to claim 1, wherein the combination of sequences comprises an anchor and a sequence adjacent to the anchor.

8. A method according to claim 1, wherein the combination of sequences comprises the first and last sequence of the section.

9. A method according to claim 1, wherein the combination of sequences comprises more than one sequence associated with an anchor,

wherein the combination of anchor and sequences to form the section is determined, and

wherein the section is only extracted when all sequences forming the section are present.

10. A method according to claim 1, wherein searches for combinations of sequences are carried out in parallel on different sections of the data stream.

11. A method according to claim 1, wherein each sequence comprises a series of bits of data, or multiple bytes of data.

12. A method according to claim 1, wherein the end point identifiers comprise one of a domain name, an email address, a uniform resource locator, a telephone number, or a date and time format.

13. A method according to claim 1, wherein each sequence is encoded in a separate state machine.

14. A method according to claim 13, wherein multiple state machines are combined to represent the section.

15. A method according to claim 14, wherein a bridge provides a transition between separate state machines representing the sequences of the section

16. A method according to claim 1, the method further comprising:

filtering the extracted sections of the data stream, the filtering comprising determining a set of characters of interest;

testing each section of the data stream for the presence of one or more of the set of characters of interest; and

extracting sections in which at least one of the characters is present.

17. A method according to claim 16, comprising:

determining a further set of characters of interest;

testing for at least one character from the further set of characters in the portion of the data stream; and

extracting sections in which at least one of the characters from the further sets of characters is also present in the section.

18. A method according to claim 16, wherein the extracted sections are stored in a store.

19. A method according to claim 16, wherein the extracted sections are

input to a comparison stage;

compared with specific examples of end point identifiers; and

discarded when the section does not match one of the specific examples in the comparison stage.

20. The method of claim 19, the determining, testing and extraction are performed by an extraction and filtering server that is a stand-alone server, a component of another server, a component of a terminal or a component of a client.

21. The method of claim 20, comprising, wherein when extracted sections match one of the specific examples in the comparison stage then data associated with the extracted sections is not passed to a client and when extracted section do not match one of the specific examples in the comparison stage then data associated with the extracted section is passed to the client.

22. The method of claim 20, comprising, wherein when extracted sections does not match one of the specific examples in the comparison stage then data associated with the extracted sections is not passed to a client and when extracted section matches one of the specific examples in the comparison stage then data associated with the extracted section is passed to the client.

23. The method of claim 22, wherein the data associated with the extracted section is an email or a web page.