Methods and apparatus for user-centered web crawling

• US 20040205049A1
• Filed: 04/10/2003
• Published: 10/14/2004
• Est. Priority Date: 04/10/2003
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Aggarwal, Charu C.
• Application Number:
  US10/410,846
• Publication Number:
  US 20040205049A1
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

Techniques are provided for user-centered search and crawling on an information network such as the world wide web. The techniques identify the nature of the web pages which are most relevant to a given predicate. The behavior of users is used to identify and determine the web pages which are most relevant to a specific crawl. Thus, the techniques are implemented in a web crawling system which can obtain the web pages specific to a given topic by leveraging the nature of the interests of the users in different topics.

Claims

1. A computer-based method of performing document retrieval in accordance with an information network, the method comprising the steps of:

obtaining a query comprising at least a user-defined predicate;

determining a group of one or more users for a set of one or more documents that satisfy the predicate, the user group comprising one or more users who have previously accessed at least one of the one or more documents in the set, wherein a determination of whether a user has previously accessed a document is obtained from a log that maintains data representing user document access behavior;

determining a topical inclination value for each user in the user group, the topical inclination value for each user being indicative of a level of interest the user has in the one or more documents in the set;

determining a topical affinity value for each document accessed by the user group based on the topical inclination value determined for each user, the topical affinity value for each document being indicative of the likelihood that each document satisfies the predicate based on the access behavior associated with the one or more users in the user group; and

outputting the one or more documents ranked in accordance with their respective topical affinity values.

2. The method of claim 1, wherein the log data comprises data representing user access frequency of documents previously accessed.

3. The method of claim 1, wherein the log data comprises data representing a topical distribution of documents previously accessed.

4. The method of claim 1, wherein the step of determining the topical inclination value for a user comprises utilizing a predicate satisfaction percentage of the one or more documents accessed by a user to determine a level of inclination of that user to topics associated with the one or more documents.

5. The method of claim 1, wherein the user document access behavior is obtained from traces.

6. The method of claim 1, wherein the topical inclination value for each user is defined by an access frequency of the one or more documents belonging to the predicate compared to all other documents.

7. The method of claim 1, wherein time spent by a user on a document is used to determine the topical inclination value for the user.

8. The method of claim 1, wherein the topical inclination values of users accessing a document are averaged to determine the topical affinity value of the document.

9. Apparatus for performing document retrieval in accordance with an information network, the apparatus comprising:

a memory; and

at least one processor coupled to the memory and operative to;

(i) obtain a query comprising at least a user-defined predicate;

(ii) determine a group of one or more users for a set of one or more documents that satisfy the predicate, the user group comprising one or more users who have previously accessed at least one of the one or more documents in the set, wherein a determination of whether a user has previously accessed a document is obtained from a log that maintains data representing user document access behavior;

(iii) determine a topical inclination value for each user in the user group, the topical inclination value for each user being indicative of a level of interest the user has in the one or more documents in the set;

(iv) determine a topical affinity value for each document accessed by the user group based on the topical inclination value determined for each user, the topical affinity value for each document being indicative of the likelihood that each document satisfies the predicate based on the access behavior associated with the one or more users in the user group; and

(v) output the one or more documents ranked in accordance with their respective topical affinity values.

10. The apparatus of claim 9, wherein the log data comprises data representing user access frequency of documents previously accessed.

11. The apparatus of claim 9, wherein the log data comprises data representing a topical distribution of documents previously accessed.

12. The apparatus of claim 9, wherein the operation of determining the topical inclination value for a user comprises utilizing a predicate satisfaction percentage of the one or more documents accessed by a user to determine a level of inclination of that user to topics associated with the one or more documents.

13. The apparatus of claim 9, wherein the user document access behavior is obtained from traces.

14. The apparatus of claim 9, wherein the topical inclination value for each user is defined by an access frequency of the one or more documents belonging to the predicate compared to all other documents.

15. The apparatus of claim 9, wherein time spent by a user on a document is used to determine the topical inclination value for the user.

16. The apparatus of claim 9, wherein the topical inclination values of users accessing a document are averaged to determine the topical affinity value of the document.

17. An article of manufacture for performing document retrieval in accordance with an information network, comprising a machine readable medium containing one or more programs which when executed implement the steps of:

obtaining a query comprising at least a user-defined predicate;

determining a group of one or more users for a set of one or more documents that satisfy the predicate, the user group comprising one or more users who have previously accessed at least one of the one or more documents in the set, wherein a determination of whether a user has previously accessed a document is obtained from a log that maintains data representing user document access behavior;

determining a topical inclination value for each user in the user group, the topical inclination value for each user being indicative of a level of interest the user has in the one or more documents in the set;

determining a topical affinity value for each document accessed by the user group based on the topical inclination value determined for each user, the topical affinity value for each document being indicative of the likelihood that each document satisfies the predicate based on the access behavior associated with the one or more users in the user group; and

outputting the one or more documents ranked in accordance with their respective topical affinity values.

18. The article of claim 17, wherein the log data comprises data representing user access frequency of documents previously accessed.

19. The article of claim 17, wherein the log data comprises data representing a topical distribution of documents previously accessed.

20. The article of claim 17, wherein the step of determining the topical inclination value for a user comprises utilizing a predicate satisfaction percentage of the one or more documents accessed by a user to determine a level of inclination of that user to topics associated with the one or more documents.

21. The article of claim 17, wherein the user document access behavior is obtained from traces.

22. The article of claim 17, wherein the topical inclination value for each user is defined by an access frequency of the one or more documents belonging to the predicate compared to all other documents.

23. The article of claim 17, wherein time spent by a user on a document is used to determine the topical inclination value for the user.

24. The article of claim 17, wherein the topical inclination values of users accessing a document are averaged to determine the topical affinity value of the document.