Plug-in parsers for configuring search engine crawler

• US 8,527,495 B2
• Filed: 02/19/2002
• Issued: 09/03/2013
• Est. Priority Date: 02/19/2002
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Redpath, Richard J.
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US10/078,784
• Publication Number:
  US 20030158835A1
• Time in Patent Office: 4,214 Days
• Field of Search: 707 1- 10, 707100-1041, 707/200, 707709-710, 707/706, 707/748, 709203-226, 705 26- 27, 705 1- 2, 705/10, 715512-513, 713200-201, 706 45- 50
• US Class Current: 707/709
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/95 Retrieval from the web

Abstract

A plug-in interface is provided in a crawling search engine. Plug-in parsers are also provided for use with the search engine. The plug-in interface allows the search engine to be configured with different plug-in parsers. Thus, a customer may configure a search engine with a parser that best suits the needs of the customer and to try new parsing algorithms to find the best results.

Claims

1. A method in a recordable-type media for configuring a search engine, the method comprising:

providing a plug-in interface for a crawling search engine; and

providing a plurality of different plug-in parsers for use with the crawling search engine,

wherein the plug-in interface allows the crawling search engine to be configured with different plug-in parsers,

wherein the plurality of different plug-in parsers parse page data for window chunks using different parsing algorithms.

2. The method of claim 1, wherein the crawling search engine recursively crawls anchors in the window chunks.

3. The method of claim 2, wherein the crawling search engine determines edges of the recursive crawling and determines weights for page data of the edges.

4. The method of claim 3, wherein the crawling search engine compile s a final results list based on the weights.

5. The method of claim 1, wherein each plug-in parser of the plurality of different plug-in parsers includes a weighting algorithm.

6. A method in a recordable-type media for searching documents, the method comprising:

loading a plug-in parser by a crawling search engine from a plurality of different plug-in parsers for use with the crawling search engine via a plug-in interface for the crawling search engine, wherein the plurality of different plug-in parsers parses page data for window chunks using different parsing algorithms;

obtaining a list of pages;

obtaining page data for the pages;

using the plug-in parser to parse the page data for a set of window chunks; and

recursively crawling anchors in the set of window chunks.

7. The method of claim 6, wherein the step of recursively crawling anchors comprises:

determining edges of the recursive crawling; and

determining weights for page data of the edges.

8. The method of claim 7, further comprising:

compiling a results list based on the weights.

9. The method of claim 6, wherein the plug-in parser is a first plug-in parser, the list of pages is a first list of pages, the page data is first page data, and the set of window chunks is a first set of window chunks, the method further comprising:

loading a second plug-in parser;

obtaining a second list of pages;

obtaining second page data for the second list of pages;

using the second plug-in parser to parse the second page data for a second set of window chunks; and

recursively crawling anchors in the second set of window chunks.

10. An apparatus for searching documents, comprising:

a crawling search engine, wherein the crawling search engine includes a plug-in interface that allows the crawling search engine to be configured with different plug-in parsers;

a memory that contains a set of instructions;

a plurality of different plug-in parsers for use with the crawling search engine, wherein the plurality of different plug-in parsers parse page data for window chunks using different parsing algorithms; and

a processing unit, responsive to execution of the set of instructions, loading from the crawling search engine one of the the plurality of different plug-in parsers via the plug-in interface and parsing with the loaded plug-in parser page data for window chunks.

11. The apparatus of claim 10, wherein the crawling search engine recursively crawls anchors in the window chunks.

12. The apparatus of claim 11, wherein the crawling search engine determines edges of the recursive crawling and determines weights for page data of the edges.

13. The apparatus of claim 12, wherein the crawling search engine compiles a final results list based on the weights.

14. The apparatus of claim 10, wherein each plug-in parser of the at least one plug-in parser includes a weighting algorithm.

15. A recordable-type medium stored thereon computer usable program code for configuring a search engine, the computer usable program code, when executed by a computer, causes the computer to perform:

providing a plug-in interface for a crawling search engine; and

providing a plurality of different plug-in parsers for use with the crawling search engine,

wherein the plug-in interface allows the crawling search engine to be configured with different plug-in parsers,

wherein the plurality of different plug-in parsers parse page data for window chunks using different parsing algorithms.

16. The recordable-type medium of claim 15, wherein the crawling search engine includes instructions for recursively crawling anchors in the window chunks.

17. A recordable-type medium stored thereon computer usable program code for searching documents, the computer usable program code, when executed by a computer, causes the computer to perform:

loading a plug-in parser by a crawling search engine from a plurality of different plug-in parsers for use with the crawling search engine via a plug-in interface for the crawling search engine, wherein the plurality of different plug-in parsers parse page data for window chunks using different parsing algorithms;

obtaining a list of pages;

obtaining page data for the pages;

using the plug-in parser to parse the page data for window chunks; and

recursively crawling anchors in the window chunks.

18. A method in a recordable-type media for configuring a search engine, the method comprising:

providing a plug-in interface for a crawling search engine;

providing a plurality of different plug-in parsers for use with the crawling search engine;

wherein the plug-in interface allows the crawling search engine to be configured with different plug-in parsers,

wherein the plurality of different plug-in parsers parse page data for window chunks using different parsing algorithms;

wherein the crawling search engine loads one of the plurality of different plug-in parsers via the plug-in interface;

wherein the loaded plug-in parser performs the step of parsing the page data before the crawling search engine performs other actions relative to the page data.