System and method for automatic generation of dynamic search abstracts contain metadata by crawler

• US 6,931,397 B1
• Filed: 02/11/2000
• Issued: 08/16/2005
• Est. Priority Date: 02/11/2000
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Sundaresan, Neelakantan
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US09/502,818
• Time in Patent Office: 2,013 Days
• Field of Search: 707 1- 10, 707100-1041, 707200-205, 707/501.1, 707/500.1, 707512-513, 709331-332, 709/312, 709218-220, 709200-203, 709228-229, 705 26- 27, 345/968, 715/501.1, 715/511, 715/513
• US Class Current: 707/707
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A computer program product is provided as an abstract update system that improves Internet searching by automatically generating accurate and dynamic search abstracts. The abstract update system enables search engines to provide the most recent abstract information for a document that matches a search query, even if this document may have been crawled after the metadata was indexed. The dynamic abstracts are generated from the latest crawl metadata and link information stored in a link repository. As a result, the dynamic abstracts contain metadata based primarily on changes to a candidate page itself, and changes to what other sources provide about this candidate page. For example, even if a candidate page has not changed, new pages may point to it; some pages that used to point to the candidate page no longer do so; or existing pages that point to the candidate page change their annotations.

Claims

1. A system for automatically generating dynamic search abstracts, comprising:

a crawler for crawling documents and acquiring metadata and link information from the documents;

a metadata repository for storing the metadata acquired by the crawler;

a link repository for storing link information acquired by the crawler;

an abstract engine for generating abstracts of the documents from the metadata;

an indexing engine for periodically indexing the metadata and the link information;

a search engine for applying a search query to the metadata indexed by the indexing engine, to generate a preliminary result set containing selected abstracts; and

wherein the search engine inquires if the link repository contains new link information about the preliminary result set, and updates the selected abstracts based on the new link information, if any, to generate the dynamic search abstracts.

2. The system according to claim 1, further including a query transformer, which, when prompted by the search query, applies a query request to the metadata and the link information indexed by the indexing engine.

3. The system according to claim 1, further including a search results transformer that transforms the dynamic search abstracts into a user browsable form.

4. The system according to claim 1, wherein the link repository stores persistent link information and maintains a crawl history.

5. The system according to claim 1, wherein at least one of the selected abstracts includes information gathered from a source other than a candidate page associated with the selected abstract.

6. A computer program product for automatically generating dynamic search abstracts, comprising:

a crawler for crawling documents and acquiring metadata and link information from the documents;

a metadata repository for storing the metadata acquired by the crawler;

a link repository for storing link information acquired by the crawler;

an abstract engine for generating abstracts of the documents from the metadata;

an indexing engine for periodically indexing the metadata and the link information;

a search engine for applying a search query to the metadata indexed by the indexing engine, to generate a preliminary result set containing selected abstracts; and

wherein the search engine inquires if the link repository contains new link information about the preliminary result set, and updates the selected abstracts based on the new link information, if any, to generate the dynamic search abstracts.

7. The computer program product according to claim 6, further including a query transformer, which, when prompted by the search query, applies a query request to the metadata and the link information indexed by the indexing engine.

8. The computer program product according to claim 6, further including a search results transformer that transforms the dynamic search abstracts into a user browsable form.

9. The computer program product according to claim 6, wherein the link repository stores persistent link information and maintains a crawl history.

10. The computer program product according to claim 6, wherein at least one of the selected abstracts includes information gathered from a source other than a candidate page associated with the selected abstract.

11. A method for automatically generating dynamic search abstracts, comprising:

crawling documents and acquiring metadata and link information from the documents;

storing the metadata acquired by the crawler in a metadata repository;

storing link information acquired by the crawler in a link repository;

generating abstracts of the documents from the metadata;

periodically indexing the metadata and the link information;

applying a search query to the metadata to generate a preliminary result set containing selected abstracts; and

inquiring if the link repository contains new link information about the preliminary result set, and updating the selected abstracts based on the new link information, if any, to generate the dynamic search abstracts.

12. The method according to claim 11, wherein updating a selected abstract includes gathering information from a source other than a candidate site associated with the selected abstract.

13. The method according to claim 11, wherein if the link repository does not contain new link information, presenting abstracts previously stored in the link repository.

14. The method according to claim 11, further including applying a query request to the metadata and the link information indexed by the indexing engine.

15. The method according to claim 11, further including transforming the dynamic search abstracts into a user browsable form.

16. The method according to claim 11, further including storing persistent link information and maintaining a crawl history in the link repository.

17. A computer program product having instruction codes for automatically generating dynamic search abstracts, comprising:

a first set of instruction codes that acquire documents and metadata and link information from the documents;

a metadata repository that store the metadata;

a link repository that store the link information;

a second set of instruction codes that generate abstracts of the documents from the metadata;

a third set of instruction codes that periodically index the metadata and the link information;

a fourth set of instruction codes that apply a search query to the metadata, to generate a preliminary result set containing selected abstracts; and

a fifth set of instruction codes that inquire if the link repository contains new link information about the preliminary result set, and that update the selected abstracts based on the new link information, if any, to generate the dynamic search abstracts.

18. The computer program product according to claim 17, wherein updating a selected abstract includes gathering information from a source other than a candidate site associated with the selected abstract.

19. The computer program product according to claim 17, wherein if the link repository does not contain new link information, presenting abstracts previously stored in the link repository.

20. The computer program product according to claim 17, further including applying a query request to the metadata and the link information indexed by the indexing engine.

21. The computer program product according to claim 17, further including transforming the dynamic search abstracts into a user browsable form.

22. The computer program product according to claim 17, further including storing persistent link information and maintaining a crawl history in the link repository.