Web scan process

• US 5,855,020 A
• Filed: 02/21/1996
• Issued: 12/29/1998
• Est. Priority Date: 02/21/1996
• Status: Expired due to Term

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Infoseek Corporation (The Walt Disney Company)
• Inventors: Kirsch, Steven T.
• Primary Examiner(s): Amsbury, Wayne
• Assistant Examiner(s): RONES, CHARLES
• Application Number:
  US08/604,584
• Time in Patent Office: 1,042 Days
• Field of Search: 395/326, 395/602, 395/793, 395/610, 395/800, 395/187.01, 395/200.36, 395/200.48, 395/200.33, 345/335, 707/5, 707/9, 702/2, 702/104, 702/10
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 61/30 Managing network names, e.g. use of aliases or nicknames name-to-address mapping H04L61/45
• CPC Class Codes: H04L 61/3005 Mechanisms for avoiding name conflicts
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

An information locator system providing for the expedient acquisition, validation and updating of information locators in a heterogenous network protocol environment. The locator system includes an information location discrimination engine coupleable to a network operating in the heterogeneous network protocol environment, a validation engine coupled to the information location discrimination engine to receive information locators and a database providing for the storage of information locators as discrete searchable resource locators. The validation engine is also connected to the data base for retrieving and storing resource locators. The validation engine provides for the autonomous interrogation of the heterogeneous network protocol environment to validate a predetermined information locator as a corresponding resource locator that is unique to the discrete searchable resource locators then stored by the database. Where a valid and inferred unique information locator is found, the validation engine provides a corresponding resource locator to the data base for subsequently searchable storage.

Claims

1. A system of autonomously maintaining a searchable database of information accessible over the Internet, said system comprising:

a) a discrimination system coupleable to the Internet to receive messages including electronic mail messages and network news messages, said discrimination processing said electronic mail and network news messages to identify embedded URLs; and

b) a validation system coupleable to the Internet, said validation system coupled to said discrimination system to receive a predetermined embedded URL, said validation system enabling an access of the Internet to retrieve Web page information associated with said predetermined embedded URL; and

c) a database for searchably storing said predetermined embedded URL in association with the Web page information associated with said predetermined embedded URL.

2. The system of claim 1 wherein said validation system determines whether said predetermined embedded URL is then stored in said database and, if so stored, whether Web page information corresponding to said predetermined embedded URL then retrieved from the Internet matches Web page information corresponding to said predetermined embedded URL then stored by said data base, and if not matching, said validation system operates to update said database to store the Web page information corresponding to said predetermined embedded URL then retrieved from the Internet.

3. The system of claim 2 wherein said validation system additionally stores change data in said database associated with said predetermined embedded URL, said validation system updating said change data to reflect the frequency of validation system match failures.

4. The system of claim 3 wherein said validation system selects said predetermined embedded URL from said database based on said change data associated with said predetermined URL, said validation system operating to determine whether Web page information corresponding to said predetermined embedded URL then retrieved from the Internet matches Web page information corresponding to said predetermined embedded URL then stored by said data base, and if not matching, said validation system operates to update said database to store the Web page information corresponding to said predetermined embedded URL then retrieved from the Internet.

5. The system of claim 4 wherein said validation system further operates to purge said predetermined embedded URL and associated Web page information from said database when said predetermined embedded URL is repeatedly determined to be invalid within a predetermined period of time.

6. A method of maintaining the currentness of a listing of URLs used to establish a searchable state of at least a portion of the Internet, said method comprising the steps of:

a) associating predetermined volatility data with a predetermined URL within a list of URLs;

b) associating predetermined contextual data with said predetermined URL; and

c) periodically validating the currentness of said predetermined URL, including

i) determining whether said predetermined volatility data corresponds to predetermined validation criteria;

ii) determining the validity of said predetermined URL;

iii) determining the currentness of said predetermined contextual data associated with said predetermined URL where said predetermined URL is determined to be valid

iv) accessing at least a portion of the Internet to update said predetermined contextual data associated with said predetermined URL where said predetermined contextual data is determined to be not current; and

iv) updating said predetermined volatility data to reflect the validity, currentness, and frequency that said predetermined contextual data is updated.

7. The method of claim 6 wherein said predetermined validation criteria is increasingly sensitive to the increasing frequency that said predetermined contextual data is determined to be not current.

8. The method of claim 7 wherein said step of periodically validating the currentness of said predetermined URL is performed with a frequency approximately equal to or greater than the update frequency reflected in said predetermined volatility data.

9. The method of claim 8 further comprising the steps of:

a) adding a new predetermined URL to said list of URLs; and

b) associating new predetermined volatility data with said new predetermined URL, wherein new predetermined contextual data is obtained from the Internet and wherein the initial frequency that said new predetermined contextual data is updated is initially set to a default value.

10. The method of claim 9 further comprising the steps of:

a) processing data from a plurality of heterogeneous data resources available via the Internet to identify said new predetermined URL;

b) determining the validity of said new predetermined URL; and

c) accessing at least a portion of the Internet to obtain said new predetermined contextual data associated with said new predetermined URL.