Web scan process

• US 6,269,370 B1
• Filed: 09/15/1998
• Issued: 07/31/2001
• Est. Priority Date: 02/21/1996
• Status: Expired due to Term

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Infoseek Corporation (The Walt Disney Company)
• Inventors: Kirsch, Steven T.
• Primary Examiner(s): Black, Thomas
• Assistant Examiner(s): RONES, CHARLES
• Application Number:
  US09/153,519
• Time in Patent Office: 1,050 Days
• Field of Search: 707/10, 707/104, 395/200.48, 709/218, 709/203, 709/205, 709/206, 709/217, 709/219, 709/245, 375/220, 455/426
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 61/30 Managing network names, e.g. use of aliases or nicknames name-to-address mapping H04L61/45
• CPC Class Codes: H04L 61/3005 Mechanisms for avoiding name conflicts
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

An information locator system providing for the expedient acquisition, validation and updating of information locators in a heterogenous network protocol environment. The locator system includes an information location discrimination engine coupleable to a network operating in the heterogeneous network protocol environment, a validation engine coupled to the information location discrimination engine to receive information locators and a database providing for the storage of information locators as discrete searchable resource locators. The validation engine is also connected to the data base for retrieving and storing resource locators. The validation engine provides for the autonomous interrogation of the heterogeneous network protocol environment to validate a predetermined information locator as a corresponding resource locator that is unique to the discrete searchable resource locators then stored by the database. Where a valid and inferred unique information locator is found, the validation engine provides a corresponding resource locator to the data base for subsequently searchable storage.

Claims

1. An information provider system for providing information to users over an internet where the internet includes resource providers that provide static and dynamic resource records, said information provider system comprising:

a) a uniform resource locator database for storing a URL and an indexed resource record corresponding to said URL;

b) an information processor for processing a user request for information corresponding to said indexed resource record by accessing said database with respect to said indexed resource record and returning said URL; and

c) an update engine that receives predetermined dynamic resource records from the internet, said update engine processing said predetermined dynamic resource records to detect an embedded URL and updating said database with said detected embedded URL and a predetermined indexed resource record corresponding to said detected embedded URL.

2. The information provider system of claim 1 wherein said update engine operates to validate said detected embedded URL, including said predetermined indexed resource record, in connection with the updating of said database.

3. The information provider system of claim 2 wherein a plurality of heterogenous transmission protocols are utilized to send said static and dynamic resource records over the internet, and wherein said dynamic resource records include transient messages provided over the internet in accordance with a first protocol and wherein said static resource records may be repeatedly retrieved over the internet in accordance with a second protocol.

4. The information provider system of claim 3 wherein said dynamic resource records may include network news and electronic mail provided over the internet and wherein said update engine includes a parsing engine for identifying URLs embedded in network news articles and electronic mail messages.

5. An information locator system providing for the acquisition and validation of information locators in a heterogenous network protocol environment, said system comprising:

a) an information location discrimination engine coupleable to a network, operating within said heterogeneous network protocol environment, for identifying information locators transported by any of a plurality of network protocols;

b) a database providing for the storage of information locators as discrete searchable resource locators; and

c) a validation engine coupled to said information location discrimination engine to receive information locators and to said database for retrieving and storing resource locators, said validation engine interrogating said heterogeneous network protocol environment to validate a predetermined information locator against a corresponding resource locator stored by said database, said validation engine providing said corresponding resource locator to said database.

6. The information locator system of claim 5 wherein said validation engine selects said corresponding resource locator for validation based on the frequency that information referenced by said predetermined information locator changes.

7. The information locator system of claim 6 wherein said database stores prior information derived from information referenced by said predetermined information locator, wherein said validation engine determines whether current information referenced by said predetermined information locator has changed in relation to said prior information, and wherein said validation engine records data reflective of the frequency of change of the information referenced by said predetermined information locator.

8. The information locator system of claim 7 wherein said prior information includes predetermined contextual information derived from information referenced by said predetermined information locator and wherein a search engine is coupled between said database and said network for receiving search requests from said network, wherein predetermined contextual information is stored by said database in relation to each resource locator stored by said database, said validation engine using said predetermined contextual information in determining whether said predetermined information locator has changed, said search engine performing searches over said predetermined contextual information to return an information locator as a product of a search request.

9. A method of autonomously collecting information references from dynamic resource data provided over the internet, said method comprising the steps of:

a) receiving predetermined dynamic resource data from a plurality of sources where each said source provides for the transport of dynamic resource data subject to a respective network data transport protocol;

b) parsing said predetermined dynamic resource data received from a first source to locate a first information reference embedded in said predetermined dynamic resource data;

c) retrieving first contextual data referenced by said first information reference from a second source; and

d) storing said first contextual data in association with said first information reference in a searchable database.

10. The method of claim 9, wherein said second source provides static resource data over the internet, further comprising the steps of:

a) parsing said predetermined static resource data received from said second source to locate a second information reference embedded in said predetermined data;

b) retrieving second contextual data referenced by said second information reference from said second source; and

c) storing said second contextual data in association with said second information reference in said searchable database.

11. The method of claim 10 wherein a prior iteration of said steps of storing said first and second contextual data store identifiable prior instances of said first and second contextual data in respective association with said first and second information references in said searchable database.

12. The method of claim 11 further comprising the steps of:

a) determining whether said first information reference is stored in said database;

b) determining whether the prior instance of said first contextual data the same as the current instance of said first contextual data; and

c) selectively updating change data associated with said first information reference reflecting whether the prior instance of said first contextual data is the same as the current instance of said first contextual data.

13. The method of claim 12 further comprising the steps of:

a) selecting said first information reference based on said change data;

b) retrieving the current instance of first contextual data referenced by said first information reference from said second source;

c) determining whether the current instance of said first contextual data is the same as the prior instance of said first contextual data; and

d ) selectively updating the prior instance of said first contextual data in said searchable database with said current instance of said first contextual data.

14. The method of claim 13 further comprising the steps of:

a) receiving a network search request;

b) searching said searchable database including said first and second contextual information for a responsive match to said network search request; and

c) returning predetermined search information derived from said first and second contextual information and associated information references as stored by said database.

15. A system for establishing searchable information from which context based searches can be performed, said system comprising:

a database that stores said searchable information, including contextual information, correlated to resource locators;

a discrimination engine, coupleable to a network to receive a dynamic data feed, for selecting information locators from said dynamic data feed and for determining corresponding resource locators;

a validation engine for validating said corresponding resource locators and for obtaining corresponding contextual information, said discrimination engine coupled to said database to permit the storage of said corresponding contextual information with said corresponding resource locators in said database.

16. The system of claim 15 wherein said dynamic data feed includes data transferred over said network utilizing multiple data transfer protocols and wherein said discrimination engine is responsive to said multiple data transfer protocols for selecting information resource locators from said dynamic data feed.

17. A system for network connection to multiple information sources for establishing searchable information from which context based searches can be performed over a network, said multiple information sources including one or more static information sources and one or more dynamic information sources that operate with heterogeneous network protocols whereby different ones of the multiple information sources employ correspondingly different ones of data transfer protocols for transfer of source information over said network, said system comprising:

a database that stores said searchable information, including contextual information, correlated to resource locators;

a discrimination engine, for coupling to one or more of said multiple information sources including coupling to a dynamic data feed of one of said dynamic information sources, for selecting information locators from source information of said dynamic data feed and determining corresponding resource locators;

a validation engine for validating said corresponding resource locators and for obtaining corresponding contextual information, said discrimination engine coupled to said database to permit the storage of said corresponding contextual information with said corresponding resource locators in said database.

18. The system of claim 17 wherein said static information sources include one or more of World Wide Web, Gopher and FTP sources and said dynamic information sources include one or more of Net News and ListServ sources.

19. The system of claim 17 wherein said corresponding resource locators are Universal Record Locators used on the World Wide Web.

20. The system of claim 17 wherein said source information of said dynamic data feed exhibits a frequency of change and said validation engine selects said corresponding resource locator for validation based on said frequency of change.

21. The system of claim 17 wherein said source information of said dynamic data feed exhibits a frequency of change, wherein said validation engine selects said corresponding resource locator for validation based on said frequency of change, wherein said database stores particular searchable information corresponding to said corresponding resource locator, wherein said validation engine determines whether current information referenced by said corresponding resource locator has changed in relation to said particular stored searchable information, and wherein said validation engine records the current information to update said particular searchable information at a frequency that is a function of said frequency of change.

22. The system of claim 17 wherein a search engine is coupled between said database and said network for receiving search requests from said network and wherein said search engine performs searches over said contextual information to return a particular resource locator as a result of a particular search request.

23. A system for internet connection to multiple information sources for establishing searchable information from which context based searches can be performed, said multiple information sources including one or more static information sources and one or more dynamic information sources that operate with heterogeneous network protocols whereby different ones of the multiple information sources employ correspondingly different ones of data transfer protocols for transfer of source information, said source information available from internet locations that are distributed and unmapped, said system comprising:

a database that stores said searchable information, including contextual information, correlated to resource locators;

a discrimination engine, for coupling to one or more of said multiple information sources including coupling to a dynamic data feed of one of said dynamic information sources, for discovering and selecting information locators from source information of said dynamic data feed and determining corresponding resource locators;

a validation engine for validating said corresponding resource locators and for obtaining corresponding contextual information so as to map said internet locations of corresponding contextual information, said discrimination engine coupled to said database to permit the storage of said corresponding contextual information with said corresponding resource locators in said database.

24. A system for internet connection to multiple information sources for establishing searchable information from which context based searches can be performed, said multiple information sources including a plurality of information sources that operate with heterogeneous network protocols whereby different ones of the multiple information sources employ correspondingly different ones of data transfer protocols for transfer of source information, said source information available from internet locations that are distributed and unmapped, said system comprising:

a database that stores said searchable information, including contextual information, correlated to resource locators;

a discrimination engine, for coupling to said multiple information sources, for discovering and selecting information locators from source information having heterogeneous network protocols and determining corresponding resource locators;

a validation engine for validating said corresponding resource locators and for obtaining corresponding contextual information so as to map said internet locations of corresponding contextual information, said discrimination engine coupled to said database to permit the storage of said corresponding contextual information with said corresponding resource locators in said database.