Network content search system and method

• US 10,298,401 B1
• Filed: 03/22/2017
• Issued: 05/21/2019
• Est. Priority Date: 03/22/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Goldberg, Matthew E.
• Primary Examiner(s): Jamshidi, Ghodrat
• Application Number:
  US15/466,642
• Time in Patent Office: 790 Days
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 63/02 for separating internal from external traffic, e.g. firewalls
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 9/3236 using cryptographic hash functions
• CPC Class Codes: H04L 9/3239 involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD

Abstract

A network traffic monitoring service provides a way to search network traffic intercepted by a network firewall while protecting the privacy of non-matching network traffic that traverses the firewall. Network traffic is parsed and processed into a set of tokens. In various implementations, the tokens may be words, HTML tags, data values, or other searchable units of information. The tokens are converted into a set of hashes, and the set of hashes is provided to the traffic monitoring service. A search authority submits a search request to the traffic monitoring service. Search terms of the search request are converted to a set of hashes to produce a hashed search request. The traffic monitoring service processes the hashed search request against the set of hashes provided by the network firewall to determine whether the network traffic represented by the set of hashes matches the search request.

Claims

1. A computer-implemented method, comprising:

intercepting, at a network firewall, a network stream between a client device and an external network;

processing the network stream into a set of tokens that are present in the network stream, the set of tokens separated by one or more delimiters;

producing a set of values by applying a transformation to a first subset of tokens of the set of tokens, where the transformation, when applied to a token of the set of tokens, produces a representative value;

obtaining a search request, the search request including a search term and a request for user information;

applying the transformation to the search term to produce a set of search values;

generating a determination that the network stream satisfies the search request based at least in part on a result of comparing the set of search values to the set of values; and

providing the determination and user information associated with the network stream to an administrative console.

2. The computer-implemented method of claim 1, further comprising:

identifying a destination network port to which the network stream is addressed; and

processing the network stream into a set of tokens based at least in part on the destination network port.

3. The computer-implemented method of claim 1, further comprising:

identifying a plurality of sets of network streams by identifying a set of network streams for individual search terms in the search request; and

determining a final set of network streams that satisfies the search request by at least applying search logic included with the search request to the plurality of sets of network streams.

4. The computer-implemented method of claim 1, wherein producing a set of values is accomplished at least in part by determining a cryptographic hash of individual tokens in the set of tokens.

5. A system, comprising:

one or more processors; and

memory retaining computer-executable instructions that, if executed, cause the system to;

convert a token in a set of token sets to a corresponding hash value to produce a set of hashed token sets that represent a set of data streams;

store information associating individual hashed token sets in the set of hashed token sets with a data stream in the set of data streams;

obtain a set of hashed search terms and an associated set of search logic;

identify, from the set of data streams, a data stream that includes at least one hashed search term of the set of hashed search terms in accordance with the associated set of search logic by at least comparing a first portion of the set of hashed search terms to a second portion of the set of hashed token sets;

receive, from an entity, a request to identify a user associated with the data stream that includes the at least one hashed search term in accordance with the associated set of search logic;

determine that the entity is authorized to perform a search; and

as a result of having determined that the entity is authorized, identify a user associated with the data stream.

6. The system of claim 5, wherein converting individual tokens in the set of token sets to the corresponding hash value is accomplished by at least determining a cryptographic hash of individual tokens in the set of token sets.

7. The system of claim 5, wherein converting individual tokens in the set of token sets to the corresponding hash value is accomplished by at least encrypting individual tokens in the set of token sets with a cryptographic key.

8. The system of claim 5, wherein:

the memory retaining computer-executable instructions further causes the one or more processors to tokenize individual data streams of the set of data streams into a set of tokens to produce the set of token sets;

tokenizing individual data streams in the set of data streams is accomplished by parsing individual data streams into a corresponding sequence of words separated by whitespace characters; and

whitespace characters include space, punctuation, carriage return, and line feed characters.

9. The system of claim 5, wherein:

a particular data stream in the set of data streams is transmitted over a logical network connection between a client computer system and an external network through a network router; and

the system stores information that associates a user of the client computer system with the particular data stream.

10. The system of claim 5, wherein:

a particular data stream in the set of data streams includes HTML information streamed over a network connection; and

tokenizing the particular data stream is accomplished at least by separating the HTML information into a corresponding set of HTML tags.

11. The system of claim 5, wherein the data stream that includes the search terms in accordance with the search logic is determined in part by:

applying an OR operation represented in the associated set of search logic by determining a union between two sets of hashed token sets; and

applying an AND operation represented in the associated set of search logic by determining an intersection between two sets of hashed token sets.

12. The system of claim 5, wherein the memory retaining computer-executable instructions that cause the one or more processors to determine that the entity is authorized to perform the search further causes the one or more processors to determine that the entity is authorized based at least in part on identifying that the entity provided the search from an administrator console.

13. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:

obtain a search request from a requester, the search request containing at least one search term, where the requester is authorized to perform the search request;

determine a cryptographic hash of the search term;

obtain a set of hashes that represent a data stream, individual hashes in the set of hashes being a cryptographic hash of a different portion of the data stream;

determine that a portion of the data stream matches the search term by at least determining that at least one hash in the set of hashes matches the cryptographic hash of the search term

obtain, from an administrator console, an identification request associated with the data stream; and

provide identifying information associated with the data stream to the administrator console.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to:

obtain information that identifies a user that is associated with the data stream; and

provide the information to the requester.

15. The non-transitory computer-readable storage medium of claim 13, wherein:

the set of hashes are received from a network firewall server; and

the data stream is a stream of data sent over a logical connection that traverses the network firewall server.

16. The non-transitory computer-readable storage medium of claim 13, wherein:

the search request includes an additional search term;

the search request specifies that a logical AND operation be performed between the search term and the additional search term; and

the instructions further include instructions that cause the computer system to;

determine an additional cryptographic hash from the additional search term; and

determine that the search request is satisfied by at least determining that a different portion of the data stream matches the additional search term by at least determining that at least one hash in the set of hashes matches the additional cryptographic hash.

17. The non-transitory computer-readable storage medium of claim 13, wherein:

the search request includes an additional search term;

the search request specifies that logical OR operation to be performed between the search term and the additional search term; and

the instructions further include instructions that cause the computer system to;

determine an additional cryptographic hash from the additional search term; and

determine that the search request is satisfied by at least determining that either a different portion of the data stream matches the additional search term by at least determining that at least one hash in the set of hashes matches the additional cryptographic hash or determining that the portion of the data stream matches the search term.

18. The non-transitory computer-readable storage medium of claim 13, wherein:

the search request specifies that a logical NOT operation be performed with respect to the search term; and

the instructions further include instructions that cause the computer system to determine that the search request is satisfied by at least determining that no portion of the data stream matches the search term.

19. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to:

obtain the data stream; and

provide the data stream to the requester.

20. The non-transitory computer-readable storage medium of claim 13, wherein the cryptographic hash is generated with a SHA-1, SHA-2, MD5, or BLAKE hashing function.