Information collection apparatus, search engine, information collection method, and program

• US 8,676,782 B2
• Filed: 08/14/2009
• Issued: 03/18/2014
• Est. Priority Date: 10/08/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hamada, Seiji, Yamamoto, Makoto
• Primary Examiner(s): TIMBLIN, ROBERT M
• Application Number:
  US13/003,875
• Publication Number:
  US 20110119263A1
• Time in Patent Office: 1,677 Days
• Field of Search: 707/726, 707/706, 707/709, 707/710, 707/711
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The present invention provides an information collection apparatus, an information collection method, and a program capable of collecting information from information resources on a network effectively as well as a search engine that searches the information resources collected. An information collection apparatus of the present invention that collects information from information resources on a network includes an extraction unit that acquires data from an information resource via the network to extract a link-destination address included in the data, a calculation unit that calculates, by comparing each link-destination address with a collection rule describing a set of addresses qualified for a collection target, a score for each link-destination address that reflects a distance from the set to a link-destination information resource indicated by the link-destination address, and a judgment unit that judges whether the link-destination information resource is to be included in the collection target or not in accordance with the score calculated for the link-destination information resource.

Claims

1. A method comprising:

providing a collection rule defining a set of qualifying addresses;

acquiring data from an information resource over a network to extract link destination addresses in the data;

determining whether the link destination addresses are included in the set of qualifying addresses by determining whether the link destination addresses comprise addresses in an allowed set explicitly allowed by the collection rule, explicitly excluded by the collection rule, and outside of the allowed set;

assigning scores to the link destination addresses based on the determination of whether the link destination addresses are in the set of qualifying addresses, which is based on the determination of whether the link destination addresses comprise addresses explicitly allowed, explicitly excluded, or outside of the allowed set;

determining from the scores of the link destination addresses whether the link destination addresses are to be included in a collection target; and

accessing data at the link destination addresses included in the collection target to store for searching.

2. The method of claim 1, wherein the qualifying addresses comprise the allowed set of addresses comprising at least one of the addresses explicitly allowed and not explicitly excluded by the collection rule.

3. The method of claim 2, wherein a first score is assigned to link destination addresses explicitly allowed by the collection rule, a second score is assigned to link destination addresses outside of the allowed set, and a third score is assigned to link destination addresses explicitly excluded by the collection rule.

4. The method of claim 3, wherein the first score is greater than the second score and the second score is greater than the third score.

5. The method of claim 2, wherein determining whether the link destination addresses are to be included in the collection target comprises determining whether the score of the link destination addresses is greater than a score threshold, wherein the link destination addresses included in the collection target include link destinations addresses allowed by the collection rule and outside of the allowed set.

6. The method of claim 2, wherein the link destination addresses are included in a first accessed page and subsequently accessed pages linked to by link destination addresses in the first accessed page and the subsequently accessed pages, wherein the subsequently accessed pages are directly or indirectly linked to the first accessed page by link destination addresses.

7. The method of claim 2, wherein the score assigned to the link destination addresses is further adjusted based on whether the link destination addresses are in data acquired from one of the link destination addresses explicitly allowed by the collection rule or in data acquired from one of the link destination addresses outside of the allowed set.

8. The method of claim 7, wherein adjusting the score comprises subtracting an amount from the score of each of the link destination addresses in data acquired from one of the link destination addresses outside of the allowed set.

9. The method of claim 7, wherein at least one of the link destination addresses comprises a multi-linked link destination address referenced in first data at a first link destination address explicitly allowed by the collection rule and in second data at a second link destination address outside of the allowed set, and wherein the score assigned to the multi-linked destination address comprises a maximum of the scores assigned based on the link destination addresses included in the first data and the second data.

10. The method of claim 2, wherein adjusting the score for one of the link destination addresses outside of the allowed set comprises:

determining a characteristic of an addressed server addressed by the link destination address; and

adjusting the score for the link destination address based on the determined characteristic of the addressed server addressed by the link destination address.

11. The method of claim 10, wherein the determined characteristic is the addressed server including addresses allowed by the collection rule, the addressed server being in an adjacent domain to a domain of the server including addresses allowed by the collection rule, and the addressed server being in a domain external to the domain of the server including addresses allowed by the collection rule.

12. The method of claim 2, wherein the operations further comprising:

assigning a time limit to each score, wherein the scores of link destination addresses whose time limit indicates that the score has expired are not included in the collection target.

13. The method of claim 2, further comprising:

in response to determining that the time limit for the score for one link destination address has expired, determining whether the expired link destination address is included in data that has not been deleted; and

recalculating the score for the link destination addresses whose time limit has expired and that are included in data that has not been deleted.

14. The method of claim 1, wherein storing the data at the link destination addresses for searching comprises including information on data at the link destination addresses in a search index to allow searching for the data at the link destination addresses, further comprising:

receiving a search request from a client;

processing the search index to determine link destination addresses having data satisfying the search request;

ranking the determined link destination addresses according to their assigned scores; and

returning a search result to the client of the determined link destination addresses ranked according to their assigned scores.

15. The method of claim 2, wherein the operations further comprising:

maintaining as candidates for items to be added to the collection rule link destination addresses outside of the allowed set whose score satisfies criteria to be included in the collection target.

16. A non-transitory computer readable recording medium including a program executed to perform operations, the operations comprising:

providing a collection rule defining a set of qualifying addresses;

acquiring data from an information resource over a network to extract link destination addresses in the data;

determining whether the link destination addresses are included in the set of qualifying addresses by determining whether the link destination addresses comprise addresses in an allowed set explicitly allowed by the collection rule, explicitly excluded by the collection rule, and outside of the allowed set;

assigning scores to the link destination addresses based on the determination of whether the link destination addresses are in the set of qualifying addresses, which is based on the determination of whether the link destination addresses comprise addresses explicitly allowed, explicitly excluded, or outside of the allowed set;

determining from the scores of the link destination addresses whether the link destination addresses are to be included in a collection target; and

accessing data at the link destination addresses included in the collection target to store for searching.

17. The computer readable recording medium of claim 16, wherein the qualifying addresses comprise the allowed set of addresses comprising at least one of the addresses explicitly allowed and not explicitly excluded by the collection rule.

18. The computer readable recording medium of claim 17, wherein adjusting the score for one of the link destination addresses outside of the allowed set comprises:

determining a characteristic of an addressed server addressed by the link destination address; and

adjusting the score for the link destination address based on the determined characteristic of the addressed server addressed by the link destination address.

19. A system in communication with a network having information resources, comprising:

a processor; and

a computer readable medium having a program executed by the processor to perform operations, the operations comprising;

providing a collection rule defining a set of qualifying addresses;

acquiring data from an information resource over a network to extract link destination addresses in the data;

determining whether the link destination addresses are included in the set of qualifying addresses by determining whether the link destination addresses comprise addresses in an allowed set explicitly allowed by the collection rule, explicitly excluded by the collection rule, and outside of the allowed set;

assigning scores to the link destination addresses based on the determination of whether the link destination addresses are in the set of qualifying addresses, which is based on the determination of whether the link destination addresses comprise addresses explicitly allowed, explicitly excluded, or outside of the allowed set;

determining from the scores of the link destination addresses whether the link destination addresses are to be included in a collection target; and

accessing data at the link destination addresses included in the collection target to store for searching.

20. The system of claim 19, wherein the qualifying addresses comprise the allowed set of addresses comprising at least one of the addresses explicitly allowed and not explicitly excluded by the collection rule.

21. The system of claim 20, wherein adjusting the score for one of the link destination addresses outside of the allowed set comprises:

determining a characteristic of an addressed server addressed by the link destination address; and

adjusting the score for the link destination address based on the determined characteristic of the addressed server addressed by the link destination address.