ADAPTIVE WEIGHTED CRAWLING OF USER ACTIVITY FEEDS

• US 20100114946A1
• Filed: 11/06/2008
• Published: 05/06/2010
• Est. Priority Date: 11/06/2008
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Sahni, Saurabh, Kumar, Mani, Kothari, Pankaj
• Granted Patent: US 8,468,158 B2
• Time in Patent Office: Days
• US Class Current: 707/770
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

Embodiments are directed towards determining frequencies for adaptively crawling user activity feeds, by taking into account user activity patterns, as well as a type or nature of a service to be crawled. A statistical model of activity patterns is generated for each user and user activity source based on various parameters. The activity patterns may be weighted to account for variation in relevancy of user content. Various measures are generated that are adaptive by being updated at a rate determined sufficient to detect user activity pattern changes. The statistical model produces clusters of users for each service. A crawling frequency may be assigned to the various clusters such that different users for a given service may have content crawled at different frequencies.

Claims

1. A network device, comprising:

a transceiver to send and receive data over a network; and

a processor that is operative to perform actions, including;

for each of a plurality of users of a user activity feed;

collecting information about the respective user'"'"'s networking activity patterns;

determining from the collected information, a user'"'"'s update score, popularity score, connection score, and quality of vitality score; and

combining the user'"'"'s update score, popularity score, connection score, and quality of vitality score to generate an adaptive overall user score; and

based on the overall user scores for the each of the plurality of users, clustering the users into a plurality of different clusters;

assigning a crawling frequency to each of the plurality of different clusters;

crawling the user activity feed for each user within a given cluster within the plurality of different clusters at the assigned crawling frequency to collect update information from the user activity feed for each user within the cluster; and

distributing to each of the plurality of users the collected update information for the respective user to display at a respective client device.

2. The network device of claim 1, wherein assigning the crawling frequency is based on a function of an average of user scores for the users within a given cluster and information about the user activity feed, including a rate limited for crawling the user activity feed.

3. The network device of claim 1, wherein clustering the users into a plurality of different clusters is performed using one of a K-means algorithm, an expectation-maximization algorithm, or a co-relation clustering algorithm.

4. The network device of claim 1, wherein the collected information includes at least one of a content generation patter for each user, a strength of a user'"'"'s connections, a popularity of each user, a popularity of a posting by each user, a popularity of a topic or tag of content generated by each user, or a cyclical activity pattern for each user.

5. The network device of claim 1, updating each of the user'"'"'s adaptive overall user scores at a frequency to detect a change in the respective user'"'"'s network activity patterns.

6. The network device of claim 1, wherein assigning the crawling frequency is based on a function of an average of overall scores for users within a cluster and at least one of a type of the user activity feed, a pattern of content generation of the user activity feed, or a rate limit for the user activity feed.

7. A system that is operative to manage a search query, comprising:

an aggregator configured to adaptively crawl a plurality of network services to retrieve changes to content at one or more of the plurality of network services, wherein at least one network service is associated with a user activity feed; and

an adaptive aggregation component having computer-executable instructions, the computer-executable instructions when installed onto a computing device enable the computing device to perform actions, comprising;

collecting information about each of a plurality of users for each of the plurality of network services to detect dynamic changes in each user'"'"'s network activity;

employing the collected information about each of the plurality of users to generate an adaptive overall user score for each user for each network service;

for each of the plurality of network services, generating clusters of users based on their adaptive overall user scores;

computing a crawling frequency for each cluster of users for each of the plurality of network services;

enabling the aggregator to crawl each of the plurality of network services based on the computed crawling frequency for each cluster of users to obtain updates to the respective network service for each of the users in the cluster; and

distributing to each user in the plurality of users the obtained updates such that each is enabled to display the distributed updates at their respective client device.

8. The system of claim 7, wherein collecting information about each of the plurality of users further comprises collecting user specific information for each network service that includes at least one of a content generation pattern for each user, a strength of a user connection, a popularity of each user, a popularity of a posting by each user, a popularity of a topic or tag of content generated by each user, or information about a cyclical activity pattern for each user.

9. The system of claim 7, wherein generating an adaptive overall user score further comprises:

computing for each user for each network service at least two user measures;

of an update score, a popularity score, a connection score, or a quality of vitality score; and

generating the adaptive overall user score based on a weighted combination of the at least two user measures for each user for each network service.

10. The system of claim 7, wherein the adaptive aggregation component enables the computing device to perform actions, further comprising:

based in part on a change in at least one user'"'"'s adaptive overall user score, reclassifying the at least one user into a different cluster for at least one network service.

11. The system of claim 7, wherein generating clusters of users based on their adaptive overall user scores further comprises using at least one of the following mechanisms to generate and cluster the users:

K-means algorithm, expectation-maximization algorithm, or a co-relation clustering mechanism.

12. A computer-readable storage medium having computer-executable instructions, the computer-executable instructions when installed onto a computing device enable the computing device to perform actions, comprising:

for each of a plurality of users of a user network service;

collecting information about the respective user'"'"'s networking activity patterns;

determining from the collected information, a user'"'"'s update score, popularity score, connection score, and quality of vitality score; and

combining the user'"'"'s update score, popularity score, connection score, and quality of vitality score to generate an adaptive overall user score; and

based on the overall user scores for the each of the plurality of users, clustering the users into a plurality of different clusters;

assigning a crawling frequency to each of the plurality of different clusters;

crawling the feed service for each user within a given cluster within the plurality of different clusters at the assigned crawling frequency to collect update information from the network service for each user within the cluster; and

distributing to each of the plurality of users the collected update information for the respective user to display at a respective client device.

13. The computer-readable storage medium of claim 12, wherein assigning the crawling frequency is based on a function of an average of overall scores for users within a cluster and at least one of a type of the network service, a pattern of content generation of the network service, or a rate limit for the network service.

14. The computer-readable storage medium of claim 12, wherein the collected information includes at least one of a content generation patter for each user, a strength of a user'"'"'s connections, a popularity of each user, a popularity of a posting by each user, a popularity of a topic or tag of content generated by each user, or a cyclical activity pattern for each user.

15. The computer-readable storage medium of claim 12, wherein clustering the users into a plurality of different clusters is performed using one of a K-means algorithm, an expectation-maximization algorithm, or a co-relation clustering algorithm.

16. The computer-readable storage medium of claim 12, based in part on a change in at least one user'"'"'s adaptive overall user score, reclassifying the at least one user into a different cluster.

17. A method of providing aggregation feed updates, comprising:

collecting information about each of a plurality of users for each of the plurality of network services to detect dynamic changes in each user'"'"'s network activity;

employing the collected information about each of the plurality of users to generate an adaptive overall user score for each user for each network service;

for each of the plurality of network services, generating clusters of users based on their adaptive overall user scores;

computing a crawling frequency for each cluster of users for each of the plurality of network services;

crawling each of the plurality of network services based on the computed crawling frequency for each cluster of users to obtain updates to the respective network service for each of the users in the cluster; and

distributing to each user in the plurality of users the obtained updates such that each is enabled to display the distributed updates at their respective client device.

18. The method of claim 17, wherein generating an adaptive overall user score further comprises:

generating the adaptive overall user score based on a weighted combination of n update score, a popularity score, a connection score, or a quality of vitality score

19. The method of claim 17, wherein collecting information is performed at least once per day, and wherein generating an adaptive overall user score for each user is performed at least once per day.

20. The method of claim 17, wherein computing the crawling frequency further comprises determining the crawling frequency based on a type of network service, a pattern of content generation detected for the network service, or a rate limit for crawling the network service.