Plug-in parsers for configuring search engine crawler

• US 20030158835A1
• Filed: 02/19/2002
• Published: 08/21/2003
• Est. Priority Date: 02/19/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Redpath, Richard J.
• Granted Patent: US 8,527,495 B2
• Time in Patent Office: Days
• US Class Current: 707/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/95 Retrieval from the web

Abstract

A plug-in interface is provided in a crawling search engine. Plug-in parsers are also provided for use with the search engine. The plug-in interface allows the search engine to be configured with different plug-in parsers. Thus, a customer may configure a search engine with a parser that best suits the needs of the customer and to try new parsing algorithms to find the best results.

Claims

1. A method for configuring a search engine, comprising:

providing a plug-in interface for a crawling search engine; and

providing at least one plug-in parser for the crawling search engine, wherein the at least one plug-in parser parses page data for window chunks and wherein the crawling search engine loads one of the at least one plug-in parser via the plug-in interface.

2. The method of claim 1, wherein the crawling search engine recursively crawls anchors in the window chunks.

3. The method of claim 2, wherein the crawling search engine determines edges of the recursive crawling and determines weights for page data of the edges.

4. The method of claim 3, wherein the crawling search engine compile s a final results list based on the weights.

5. The method of claim 1, wherein each plug-in parser of the at least on e plug-in parser includes a weighting algorithm.

6. A method for searching documents, comprising:

loading a plug-in parser;

obtaining a list of pages;

obtaining page data for the pages;

using the plug-in parser to parse the page data for a set of window chunks; and

recursively crawling anchors in the set of window chunks.

7. The method of claim 6, wherein the step of recursively crawling anchors comprises:

determining edges of the recursive crawling; and

determining weights for page data of the edges.

8. The method of claim 7, further comprising:

compiling a results list based on the weights.

9. The method of claim 6, wherein the plug-in parser is a first plug-in parser, the list of pages is a first list of pages, the page data is first page data, and the set of window chunks is a first set of window chunks, the method further comprising:

loading a second plug-in parser;

obtaining a second list of pages;

obtaining second page data for the second list of pages;

using the second plug-in parser to parse the second page data for a second set of window chunks; and

recursively crawling anchors in the second set of window chunks.

10. An apparatus for searching documents, comprising:

a crawling search engine, wherein the crawling search engine includes a plug-in interface; and

at least one plug-in parser for the crawling search engine, wherein the at least one plug-in parser parses page data for window chunks and wherein the crawling search engine loads one of the at least one plug-in parser via the plug-in interface.

11. The apparatus of claim 10, wherein the crawling search engine recursively crawls anchors in the window chunks.

12. The apparatus of claim 11, wherein the crawling search engine determines edges of the recursive crawling and determines weights for page data of the edges.

13. The apparatus of claim 12, wherein the crawling search engine compiles a final results list based on the weights.

14. The apparatus of claim 10, wherein each plug-in parser of the at least one plug-in parser includes a weighting algorithm.

15. An apparatus for searching documents, comprising:

loading means for loading a plug-in parser;

list means for obtaining a list of pages;

page means for obtaining page data for the pages;

parsing means for using the plug-in parser to parse the page data for window chunks; and

crawling means for recursively crawling anchors in the window chunks.

16. The apparatus of claim 15, wherein the crawling means comprises:

means for determining edges of the recursive crawling; and

means for determining weights for page data of the edges.

17. The apparatus of claim 16, further comprising:

means for compiling a final results list based on the weights.

18. The apparatus of claim 15, wherein the plug-in parser is a first plug-in parser, the list of pages is a first list of pages, the page data is first page data, and the set of window chunks is a first set of window chunks, the apparatus further comprising:

means for loading a second plug-in parser;

means for obtaining a second list of pages;

means for obtaining second page data for the second list of pages;

means for using the second plug-in parser to parse the second page data for a second set of window chunks; and

means for recursively crawling anchors in the second set of window chunks.

19. A computer program product, in a computer readable medium, for configuring a search engine, comprising:

instructions for providing a plug-in interface for a crawling search engine; and

instructions for providing at least one plug-in parser for the crawling search engine, wherein the at least one plug-in parser parses page data for window chunks and wherein the crawling search engine loads one of the at least one plug-in parser via the plug-in interface.

20. The computer program product of claim 19, wherein the crawling search engine includes instructions for recursively crawling anchors in the window chunks.

21. A computer program product, in a computer readable medium, for searching documents, comprising:

instructions for loading a plug-in parser;

instructions for obtaining a list of pages;

instructions for obtaining page data for the pages;

instructions for using the plug-in parser to parse the page data for window chunks; and

instructions for recursively crawling anchors in the window chunks.