Method for searching and analysing information in data networks

• US 20020032772A1
• Filed: 09/13/2001
• Published: 03/14/2002
• Est. Priority Date: 09/14/2000
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Overture Services Incorporated (Apollo Global Management, Inc.)
• Inventors: Olstad, Bjorn, Risvik, Knut Magne
• Granted Patent: US 7,093,012 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 2216/09 Obsolescence

Abstract

A method for providing searching and alerting capabilities in traffic content at access points in data networks is disclosed. Typical access points for Internet, intranet and wireless traffic are described. Traffic flow through an Internet Service Provider is used as a preferred embodiment to exemplify the data traffic used as the input source in the invention. The invention teaches how proper privacy and content filters can be applied to the traffic source. The filtered data stream from the traffic flow can be used to improve the quality of existing searching and alerting services. The invention also teaches how a cache can be developed optimized for holding fresh searchable information captured in the traffic flow. It is further disclosed how the said cache can be converted to a searchable index and either separately or in cooperation with external search indexes be used as a basis for improved search services. The invention also discloses how the traffic flow can be analyzed in order to derive added information for measuring document relevance, access similarity between documents, personalized ranking of search results, and regional differences in document accesses.

Claims

1. A method for searching and analysing the traffic content at access points in data networks, wherein the data networks are shared network resources, particularly the complete Internet or an intranet, wherein said access points particularly are Internet access points for users served by an Internet service provider, or distributed in the form of an application to a plurality of users, or a connection point for wireless users, said connection point being either a native connection point or an Internet gateway for multiple connection points, and wherein the method is characterized by comprising steps for collecting information in the form of data extracted from the information flow at said access points in the data network, indexing said collected information with a predefined maximum time delay, and searching and retrieving information from said indexed information, and by further optional steps based on an analysis of the traffic content at access points for location coding of collected information in the form of documents, measuring the similarity of documents, and alerting users in regard of new and updated information.

2. A method according to claim 1, characterized by storing collected information by caching in one or more cache means provided at one or more access points.

3. A method according to claim 2, characterized by caching collected information of an increasing set of time intervals in two or more cache means, indexing the cached information in each of said cache means repeatedly in order to generate associated search indexes for an increasing set of time intervals. and combining search results from the generated search indexes.

4. A method according to claim 2, characterized by keeping the cached information in a cache means until the content thereof has been indexed.

5. A method according to claim 4, characterized by computing the freshness of the cached information on the basis of elapsed time since the last update of a given traffic content unit.

6. A method according to claim 2, characterized by the step for collecting information including substeps for caching the traffic content respectively in order to minimize external bandwidth requirements and to keep fresh and relevant information for indexing.

7. A method according to claim 6, characterized by caching the traffic content in both respects in respectively two dedicated caches.

8. A method according to claim 2, characterized by processing the cached information repeatedly with a predefined maximum time delay for indexing the collected information and generating an index thereof.

9. A method according to claim 8, characterized by updating the index incrementally by detecting new or updated information that has previously not been observed at an access point, and adding said new or updated information to said search index.

10. A method according to claim 8, characterized by using the traffic content observed between two indexing operations on the cached information for generating a temporal search index with fresh traffic content since a last indexing operation performed of said cached information, and performing searching by combining searching in both said temporal search index and a most recent search index generated by indexing said cached information.

11. A method according to claim 1, characterized by limiting the indexing of the collected information to the subset of the traffic content that has not been discarded by a sensitivity filter, said sensitivity filter being adapted for detecting one or more of the following, viz. pornographic material, private material, particularly private material that only has been accessed by a given predefined small number of users, and material from http POST operations.

12. A method according to claim 1, characterized by the searching step including a substep for relevancy ranking or data filtering depending on a recorded time attribute of traffic content units, said recorded time attribute being one or more of the following, viz. the creation time a the traffic content unit, the last time a traffic content unit is modified, the time of a first observation of a given traffic content unit at an access point, and the time of the a last observation of the given traffic content unit at said access point.

13. A method according to claim 1, characterized by the searching step including a substep for relevancy ranking depending on access counting of individual traffic content units through an access point.

14. A method according to claim 1, wherein the searching step is implemented by at least one collaborating search engine, characterized by the searching step including substeps for dispatching search requests to said at least one collaborating search engine, collecting search result from a local traffic index, collecting search results from said at least one collaborating search engine and combining said collected search results to provide a unified result to an initial search request.

15. A method according to claim 14, wherein said collaborating search engine is an Internet search engine, characterized by indexing information crawled from Internet with said search engine.

16. A method according to claim 14, wherein the searching step is implemented by more than one collaborating search engine, characterized by providing a search engine in respectively at least one access point in the data network.

17. A method according to claim 1, characterized by the step for collecting information including substeps for detecting requested information that was inaccessible, and using information about said inaccessible information to either remove or reduce ranking of an associated entry in a search index, particularly a collaborating search index.

18. A method according to claim 1, wherein the searching step is implemented by at least one collaborating search engine, characterized by the step for collecting information including substeps for detecting new or updated information that has not previously been observed at an access point and using information about the new or updated information to augment a search index of said at least one collaborating search engine.

19. A method according to claim 1, characterized by the optional step for location coding including substeps for collecting document identifiers for requested documents, annotating said documents identifiers with spatial information about users submitting the requests, computing access statistics for at least one document including at least the number of document requests from a spatial region and the total number of requests from said spatial region, and determining which documents are most specific for a given spatial region by comparing the access statistics for said given spatial region with the corresponding access statistics for at least a second spatial region.

20. A method according to claim 19, characterized by deriving said spatial information from user data recorded by an Internet service provider.

21. A method according to claim 19, characterized by deriving said spatial information from the location of a mobile device requesting a document.

22. A method according to claim 19, characterized by selecting a spatial granularity such that a sufficient number of users is obtained within every spatial region in order to ensure that individual users cannot be identified in the access statistics.

23. A method according to claim 19, characterized by determining the document specificity by computing the likelihood that said access statistics for at least two spatial regions belongs to same statistical distribution.

24. A method according to claim 19, characterized by determining the document specificity by using a large spatial region as the statistics for a null hypothesis and computing the statistical significance of a difference test on said access statistics for a given spatial region.

25. A method according to claim 23, characterized by using a region specific scoring to provide location sensitive ranking of search results.

26. A method according to claim 24, characterized by using a region specific scoring to provide traffic dependent lists of popular region specific documents.

27. A method according to claim 26, characterized by dividing said document lists into category sublists by automatic category analysis of the document content.

28. A method according to claim 1, characterized by the optional step for measuring the similarity of documents including substeps for collecting document identifiers for the requested documents, annotating document requests such that consecutive requests from the same user can be identified, and computing a document similarity between a document “

b” and

a reference document “

a”

by comparing the number of “

b”

requests in the proximity of “

a”

requests with an average frequency of “

b”

requests.

29. A method according to claim 28, characterized by using the similarity measurement for providing a ranked list of similar documents based on any given document input.

30. A method according to claim 29, characterized by computing said ranked list by omitting documents from the same site as the input document.

31. A method according to claim 28, characterized by combining the similarity measurement either with a similarity measurement based on the document content or with a similarity measurement based on counting the number of co-citations.

32. A method according to claim 28, characterized by computing the similarity measurement for a group of document identifiers by treating all request to one of the documents in said group as a request for said “

a”

document.

33. A method according to claim 32, characterized by using a subset of a user'"'"'s web log as said group of document identifiers.

34. A method according to claim 33, characterized by taking said group of documents to reflect user preferences, and creating personalized ranking of search results based on said similarity measurement.

35. A method according to claim 33, characterized by computing document ranking on the basis of link topology analysis, said similarity measurement being used to define a link topology and associated weights.

36. A method according to claim 1, characterized by the optional step for alerting including substeps for collecting in real time information extracted from the data flow at an access point in the data network, detecting new and/or updated information that has not previously been observed at said access point, comparing said new and/or updated information with a set of user profiles, and sending alerting messages to users associated with user profiles that are triggered by said new and/or updated information.