Method, system, and program for gathering indexable metadata on content at a data repository

• US 6,959,326 B1
• Filed: 08/24/2000
• Issued: 10/25/2005
• Est. Priority Date: 08/24/2000
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Schell, David Allen, Dutta, Rabindranath, Day, Don Rutledge
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): Bruckart, Benjamin R
• Application Number:
  US09/645,386
• Time in Patent Office: 1,888 Days
• Field of Search: 709/203, 709/218, 709/217, 709/229, 709/202, 709/219, 707/3, 707/10
• US Class Current: 709/217
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

Disclosed is a method, system, and program for searching a data repository managed by a content provider to gather indexable metadata on content at addresses locations at the data repository. Settings capable of being customized by the content provider are accessed. The customized settings provide instructions on how to search the content provider'"'"'s data repository. The content of content pages at the content provider'"'"'s data repository is accessed in accordance with instructions included in the accessed customized settings. Metadata from accessed content pages is generated and added to an index of metadata for accessed addressable locations at the data repository.

Claims

1. A method for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, comprising:

accessing customizable settings capable of being customized by the content provider, wherein the customizable settings provide an addressable location of at least one content page in the data repository and a parameter comprising a query term for at least one included addressable location;

for each provided addressable location for which there is the query term, accessing the content page at the addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

processing the accessed content page using the provided query term at each provided addressable location to obtain query results; and

generating metadata from the obtained query results to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

2. The method of claim 1, wherein the customizable settings include parameters and access methods unique to an arrangement of content in the content provider'"'"'s data repository, wherein the access methods process the accessed content page using the parameter.

3. The method of claim 1, wherein the addressable locations comprise uniform resource locator (URL) addresses.

4. The method of claim 1, wherein the accessed customizable settings further provide qualifiers for at least one query term further comprising:

for each query term having at least one qualifier, determining whether the query results for the query term satisfy each qualifier for the query term, wherein the metadata for the query result is generated if the query result satisfies each qualifier for the query term that generated the query result; and

performing a non-qualifying action for each query result that does not satisfy each qualifier.

5. The method of claim 4, wherein the non-qualifying action comprises not including metadata for the query result in the index.

6. The method of claim 1, wherein the parameters in the accessed customizable settings further include a recursive search setting indicating whether to search hypertext links to linked addressable locations included in the accessed content page, further comprising:

accessing a content page at each linked addressable location included if the recursive search setting indicates to recursively search linked addressable locations, wherein metadata is generated for each content page recursively accessed at the linked addressable locations in the accessed content page.

7. The method of claim 6, wherein the accessed customizable settings further provide prohibited addressable locations at the data repository, wherein metadata is not generated for each content page at a linked addressable location that is one indicated prohibited address location.

8. The method of claim 1, further comprising:

determining a format of the accessed content page;

selecting one of a plurality of parsers capable of parsing the determined format; and

parsing the content page using the selected parser, wherein the metadata to add to the index is generated from the parsed content page.

9. The method of claim 1, further comprising:

determining a parser capable of parsing an embedded file referenced in the content page;

parsing the content of the referenced embedded file; and

generating metadata for the parsed content of the embedded file to add to the index.

10. The method of claim 9, wherein the embedded file is encoded in a multimedia format.

11. The method of claim 1, further comprising:

distributing a collection tool to content providers capable of accessing and generating metadata for content provider data repositories using the accessed customizable settings; and

collecting metadata gathered from multiple content providers using the collection tool to gather metadata on their data repositories.

12. The method of claim 11, further comprising commercializing the collected metadata.

13. The method of claim 11, further comprising:

receiving an electronic subscription from content providers to use the collection tool and provide metadata.

14. A method for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, comprising:

accessing customizable settings capable of being customized by the content provider, wherein the customizable settings provide an addressable location of at least one content page in the data repository and parameters for the addressable locations, wherein the parameters in the accessed customizable settings further include a password for at least one provided addressable location;

accessing the content page at the addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

using the provided password to access the content page at the indicated addressable location for which the password is provided;

processing the accessed content page using the parameter provided for the addressable location of the accessed content page to generate output information; and

generating metadata indicating the generated output information for the accessed content page to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

15. A method for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, comprising:

accessing customizable settings capable of being customized by the content provider, wherein the customizable settings provide an addressable location of at least one content page in the data repository and further indicate validation checking programs;

accessing the content page at the addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

executing each validation checking program indicated in the accessed customizable settings against each accessed content page;

generating a validation output result with the validation checking program for each accessed content page with each validation checking program describing characteristics of the content page, and

generating metadata from the validation output result to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

16. The method of claim 15, wherein the accessed customizable settings further indicate at least one parameter to use with at least one validation checking program, further comprising:

using the at least one parameter when executing the validation checking program, wherein the validation output result further indicates characteristics of the content page related to the at least one parameter used with the validation checking program.

17. The method of claim 15, wherein the parameters in the accessed customizable settings further indicate at least one qualifier to use with at least one validation checking program, further comprising:

determining whether the validation output result satisfies the at least one qualifier provided with the validation checking program producing the output result, wherein metadata for the output result is included in the index if the output result satisfies the qualifier.

18. The method of claim 17, wherein metadata for the content page at the addressable location is not included in the index if the validation output result does not satisfy the qualifier.

19. A system for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, comprising:

means for accessing customizable settings capable of being customized by the content provider, wherein the customized settings provide an addressable location of at least one content page in the data repository and a parameter in the accessed customizable settings comprising a query term for at least one included addressable location;

means for accessing the content page at each provided addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

means for processing each accessed content page using the provided query term at the provided addressable location to obtain query results for each provided addressable location for which there is a query term; and

means for generating metadata from the obtained query results to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

20. The system of claim 19, wherein the customizable settings include parameters and access methods unique to an arrangement of content in the content provider'"'"'s data repository, wherein the access methods process the accessed content page using the parameter.

21. The system of claim 19, wherein the addressable locations comprise uniform resource locator (URL) addresses.

22. The system of claim 19, wherein the accessed customizable settings further provide qualifiers for at least one query term, further comprising:

means for determining whether the query results for the query term satisfy each qualifier for the query term for each query term having at least one qualifier, wherein the metadata for the query result is generated if the query result satisfies each qualifier for the query term that generated the query result; and

means for performing a non-qualifying action for each query result that does not satisfy each qualifier.

23. The system of claim 22, wherein the non-qualifying action comprises not including metadata for the query result in the index.

24. The system of claim 19, wherein the parameters in the accessed customizable settings further include a recursive search setting indicating whether to search hypertext links to linked addressable locations included in the accessed content page, further comprising:

means for accessing a content page at each linked addressable location included if the recursive search setting indicates to recursively search linked addressable locations, wherein metadata is generated for each content page recursively accessed at the linked addressable locations in the accessed content page.

25. The system of claim 24, wherein the accessed customizable settings further provide prohibited addressable locations at the data repository, wherein metadata is not generated for each content page at a linked addressable location that is one indicated prohibited address location.

26. The system of claim 19, further comprising:

means for determining a format of the accessed content page;

means for selecting one of a plurality of parsers capable of parsing the determined format, and

means for parsing the content page using the selected parser, wherein the metadata to add to the index is generated from the parsed content page.

27. The system of claim 19, further comprising:

means for determining a parser capable of parsing an embedded file referenced in the content page;

means for parsing the content of the referenced embedded file; and

means for generating metadata for the parsed content of the embedded file to add to the index.

28. The system of claim 27, wherein the embedded file is encoded in a multimedia format.

29. The system of claim 19, further comprising:

means for distributing a collection tool to content providers capable of accessing and generating metadata for content provider data repositories using the accessed customizable settings; and

means for collecting metadata gathered from multiple content providers using the collection tool to gather metadata on their data repositories.

30. A system for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, comprising:

means for accessing customizable settings capable of being customized by the content provider, wherein the customized settings provide an addressable location of at least one content page in the data repository and parameters for the addressable locations, wherein the parameters in the accessed customizable settings further include a password for at least one provided addressable location;

means for accessing the content page at the addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

means for using the provided password to access the content page at the indicated addressable location for which the password is provided;

means for processing the accessed content page using the parameter provided for the addressable location of the accessed content page to generate output information; and

means for generating metadata indicating the generated output information for the accessed content page to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

31. A system for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, comprising:

means for accessing customizable settings capable of being customized by the content provider, wherein the customized settings provide an addressable location of at least one content page in the data repository and further indicate validation checking programs;

means for accessing the content page at the addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

means for executing each validation checking program indicated in the accessed customizable settings against each accessed content page;

means for generating a validation output result with the validation checking program for each accessed content page with each validation checking program describing characteristics of the content page; and

means for generating metadata from the validation output result to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

32. The system of claim 31, wherein the accessed customizable settings further indicate at least one parameter to use with at least one validation checking program, further comprising:

means for using the at least one parameter when executing the validation checking program, wherein the validation output result further indicates characteristics of the content page related to the at least one parameter used with the validation checking program.

33. The system of claim 31, wherein the accessed customizable settings further indicate at least one qualifier to use with at least one validation checking program, further comprising:

means for determining whether the validation output result satisfies the at least one qualifier provided with the validation checking program producing the output result, wherein metadata for the output result is included in the index if the output result satisfies the qualifier.

34. The system of claim 33, wherein metadata for the content page at the addressable location is not included in the index if the validation output result does not satisfy the qualifier.

35. A program for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, wherein the program comprises code implemented in a computer readable medium capable of causing a computer to perform:

accessing customizable settings capable of being customized by the content provider, wherein the customizable settings provide an addressable location of at least one content page in the data repository and a parameter in the accessed customizable settings comprising a query term for at least one included addressable location;

for each provided addressable location for which there is a query term, accessing the content page at the addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

processing the accessed content page using the provided query term at each provided addressable location to obtain query results; and

generating metadata from the obtained query results to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

36. The program of claim 35, wherein the customized settings include parameters and access methods unique to an arrangement of content in the content provider'"'"'s data repository, wherein the access methods process the accessed content page using the parameter.

37. The program of claim 35, wherein the addressable locations comprise uniform resource locator (URL) addresses.

38. The program of claim 35, wherein the accessed customizable settings further provide qualifiers for at least one query term, wherein the program is further capable of causing the computer to perform:

for each query term having at least one qualifier, determining whether the query results for the query term satisfy each qualifier for the query term, wherein the metadata for the query result is generated if the query result satisfies each qualifier for the query term that generated the query result; and

performing a non-qualifying action for each query result that does not satisfy each qualifier.

39. The program of claim 38, wherein the non-qualifying action comprises not including metadata for the query result in the index.

40. The program of claim 35, wherein the parameters in the accessed customizable settings further include a recursive search setting indicating whether to search hypertext links to linked addressable locations included in the accessed content page, wherein the program is further capable of causing the computer to perform:

accessing a content page at each linked addressable location included if the recursive search setting indicates to recursively search linked addressable locations, wherein metadata is generated for each content page recursively accessed at the linked addressable locations in the accessed content page.

41. The program of claim 40, wherein the accessed customizable settings further provide prohibited addressable locations at the data repository, wherein metadata is not generated for each content page at a linked addressable location that is one indicated prohibited address location.

42. The program of claim 35, wherein the program is further capable of causing the computer to perform:

determining a format of the accessed content page;

selecting one of a plurality of parsers capable of parsing the determined format; and

parsing the content page using the selected parser, wherein the metadata to add to the index is generated from the parsed content page.

43. The program of claim 35, wherein the program is further capable of causing the computer to perform:

determining a parser capable of parsing an embedded file referenced in the content page;

parsing the content of the referenced embedded file; and

generating metadata for the parsed content of the embedded file to add to the index.

44. The program of claim 43, wherein the embedded file is encoded in a multimedia format.

45. The program of claim 35, further comprising:

distributing the program to content providers capable of accessing and generating metadata for content provider data repositories using the accessed customizable settings; and

collecting metadata gathered from multiple content providers using the collection tool to gather metadata on their data repositories.

46. The program of claim 45, further comprising:

receiving an electronic subscription from content providers to use the program to gather and provide metadata.

47. A program for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, wherein the program comprises code implemented in a computer readable medium capable of causing a computer to perform:

accessing customizable settings capable of being customized by the content provider, wherein the customizable settings provide an addressable location of at least one content page in the data repository and parameters for the addressable locations, wherein the parameters in the accessed customizable settings further include a password for at least one provided addressable location;

accessing the content page at the addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

using the provided password to access the content page at the indicated addressable location for which the password is provided;

processing the accessed content page using the parameter provided for the addressable location of the accessed content page to generate output information; and

generating metadata indicating the generated output information for the accessed content page to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

48. A program for searching data repositories managed by different content providers to gather indexable metadata on content at addressable locations at the data repositories, wherein the program comprises code implemented in a computer readable medium capable of causing a computer to perform:

accessing customizable settings capable of being customized by the content provider, wherein the customizable settings provide an addressable location of at least one content page in the data repository and further indicate validation checking programs;

accessing the content page at the addressable location at the content provider'"'"'s data repository indicated in the customizable settings;

executing each validation checking program indicated in the accessed customizable settings against each accessed content page;

generating a validation output result with the validation checking program for each accessed content page with each validation checking program describing characteristics of the content page; and

generating metadata from the validation output result to add to an index of metadata for accessed addressable locations at the data repository, wherein customizable settings from the different content providers are accessed to generate the metadata for the accessed content pages, and wherein the index of metadata includes metadata on content from the data repositories of the different content providers.

49. The program of claim 48, wherein the accessed customizable settings further indicate at least one parameter to use with at least one validation checking program, wherein the program is further capable of causing the computer to perform:

using the at least one parameter when executing the validation checking program, wherein the validation output result further indicates characteristics of the content page related to the at least one parameter used with the validation checking program.

50. The program of claim 48, wherein the accessed customizable settings further indicate at least one qualifier to use with at least one validation checking program, wherein the program is further capable of causing the computer to perform:

determining whether the validation output result satisfies the at least one qualifier provided with the validation checking program producing the output result, wherein metadata for the output result is included in the index if the output result satisfies the qualifier.

51. The program of claim 50, wherein metadata for the content page at the addressable location is not included in the index if the validation output result does not satisfy the qualifier.