Thorough search of document database containing compressed and noncompressed documents

• US 6,105,021 A
• Filed: 11/21/1997
• Issued: 08/15/2000
• Est. Priority Date: 11/21/1997
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Berstis, Viktors
• Primary Examiner(s): Amsbury, Wayne
• Assistant Examiner(s): PARDO, THUY N
• Application Number:
  US08/976,213
• Time in Patent Office: 998 Days
• Field of Search: 704/9, 707/3, 707/5, 707/10, 707/104, 707/4, 395/200.34, 341/51
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A thorough search of a distributed document database, such as the Internet, is conducted according to a search request including search parameters and a search starting point. Initially, documents found at the search starting point are added to a processing list. A determination is made whether a next document in the processing list is a plain text document or a compressed document. If the next document is a compressed document, it is decompressed to produce a plain text version of the next document. Next, the plain text version of the next document is searched according to the search parameters and for references to other documents. The documents referenced by the next document are appended to the processing list. This process is repeated until inspection of the processing list is declared to be concluded. A list of processed documents which meet the search criteria are returned to a user.

Claims

1. A method for a thorough search of a distributed document database, comprising the steps of:

receiving a search request including search parameters and a search starting point;

adding a document found at the search starting point to a processing list;

determining whether a next document in the processing list is a plain text document or a compressed document;

if the next document is a compressed document, decompressing the next document to produce a plain text version of the next document;

searching the plain text version of the next document according to the search parameters and for references to other documents;

appending documents referenced by the next document to the processing list;

repeating the determining, decompressing, searching and appending steps until inspection of the processing list is concluded; and

returning a list of processed documents which meet the search criteria to a user.

2. The method as recited in claim 1 wherein inspection of the processing list is concluded according to a distance factor.

3. The method as recited in claim 2 wherein the distance factor is a number of servers from the original starting point from which documents can be retrieved.

4. The method as recited in claim 2 wherein the distance factor is a number of server switches allowed from the original starting point.

5. The method as recited in claim 1 wherein inspection of the processing list is concluded according to a time factor.

6. The method as recited in claim 1 wherein inspection of the processing list is concluded according to a resource factor.

7. The method as recited in claim 1 wherein the method is performed at a web server and the list of processed documents which meet the search criteria is returned to a web client.

8. The method as recited in claim 1 further comprising the step of caching copies of processed documents which meet the search criteria in anticipation of a request from the user for selected ones of the processed documents.

9. The method as recited in claim 1 wherein the next document is a web document and the references found to the next documents are hyperlinks to another web document.

10. A system including processor and memory for a thorough search of a distributed document database, comprising:

means for receiving a search request including search parameters and a search starting point;

means for adding a document found at the search starting point to a processing list;

means for determining whether a next document in the processing list is a plain text document or a compressed document;

means responsive to finding that the next document is a compressed document for decompressing the next document to produce a plain text version of the next document;

means for searching the plain text version of the next document according to the search parameters and for references to other documents;

means for appending documents referenced by the next document to the processing list;

means for returning a list of processed documents which meet the search criteria to a user; and

means for determining that inspection of the processing list is concluded.

11. The system as recited in claim 10 wherein inspection of the processing list is concluded according to a distance factor.

12. The system as recited in claim 11 wherein the distance factor is a number of servers from the original starting point from which documents can be retrieved.

13. The system as recited in claim 10 wherein inspection of the processing list is concluded according to a time factor.

14. The system as recited in claim 10 wherein the system a web server and the list of processed documents which meet the search criteria is returned to a web client.

15. The system as recited in claim 10 further comprising a cache for caching copies of processed documents which meet the search criteria in anticipation of a request from the user for selected ones of the processed documents.

16. A method for a thorough search of a hyperlinked document database, comprising the steps of:

receiving a search request including search parameters and a search starting point;

adding documents found at the search starting point and within a distance factor to the search starting point to a processing list;

if a respective document in the processing list is a compressed document, decompressing the document to produce a plain text version of the document;

searching the documents according to the search parameters and for references to other documents to append to the processing list; and

returning a list of processed documents which meet the search criteria to a user.

17. The method as recited in claim 16 wherein the distance factor is a number of servers from the original starting point from which documents can be retrieved.

18. The method as recited in claim 16 wherein the distance factor is a number of server switches allowed from the original starting point.

19. The method as recited in claim 16 wherein the distance factor is a number of hyperlinks traveled from a document at the original starting point.

20. A computer program product in a computer readable medium for a thorough search of a hyperlinked document database, comprising:

means for receiving a search request including search parameters and a search starting point;

means for adding documents found at the search starting point and within a distance factor to the search starting point to a processing list;

means for decompressing documents in the processing list as needed to produce plain text versions of the documents;

means for searching plain text versions of the documents according to the search parameters and for references to other documents to append to the processing list; and

means for returning a list of processed documents which meet the search criteria to a user.

21. The product as recited in claim 20 wherein the distance factor is a number of servers from the original starting point from which documents can be retrieved.

22. The product as recited in claim 20 wherein the distance factor is a number of server switches allowed from the original starting point.

23. The product as recited in claim 20 wherein the distance factor is a number of hyperlinks traveled from a document at the original starting point.