User trustworthiness

• US 9,519,682 B1
• Filed: 05/26/2011
• Issued: 12/13/2016
• Est. Priority Date: 05/26/2011
• Status: Active Grant

• Current Assignee: Yahoo Assets LLC
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Pujara, Jay, Ramarao, Vishwanath Tumkur, Shue, Jyh-Shin Gareth, Xi, Xiaopeng, Zinkevich, Martin, Dasgupta, Anirban, Tseng, Belle, Chu, Wei
• Primary Examiner(s): Edwards, Linglan
• Assistant Examiner(s): Gracia, Gary
• Application Number:
  US13/117,037
• Time in Patent Office: 2,028 Days
• Field of Search: 726/22, 709/206, 709/209
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04H 20/28 Arrangements for simultaneous broadcast of plural pieces of information
• CPC Class Codes: H04H 60/82 the transmission system being the Internet

Abstract

Embodiments are directed towards generating a unified user account trustworthiness system through user account trustworthiness scores. A trusted group of user accounts may be identified for a given action by grouping a plurality of user accounts into tiers based on a trustworthiness score of each user account for the given action. The tiers and/or trustworthiness scores may be employed to classify an item, such as a message as spam or non-spam, based on input from the user accounts. The trustworthiness scores may also be employed to determine if a user account is a robot account or a human account. The trusted group for a given action may dynamically evolve over time by regrouping the user accounts based on modified trustworthiness scores. A trustworthiness score of an individual user account may be modified based on input received from the individual user account and input from other user accounts.

Claims

1. A method comprising:

identifying, for a given action by one or more network computing devices, a trusted group of user accounts from a plurality of user accounts using a respective trustworthiness score for the given action assigned to each user account of the plurality, wherein the given action refers to an online activity that is performed by one or more users associated with respective user accounts, each user account'"'"'s trustworthiness score being determined using inputs received from the each user account for the given action, the trustworthiness score being used to identify the trusted group of user accounts whose input is to be used to classify an item, wherein the item refers to an article on the internet that can have an action performed on it, and to identify other user accounts whose input is to be excluded from classifying the item, the identifying comprising generating an initial trustworthiness score for a user account of the plurality using a trained trustworthiness classifier and a feature set about the user account, the trained trustworthiness classifier comprising a number of machine-implemented algorithms used to evolve behaviors based on input data, the feature set comprising online user behavioral features and static profile features about the user account, wherein the online user behavioral features comprise online activity features and the static profile features comprise user registration features, the initial trustworthiness score for the user account is used at least initially to determine whether or not to include the user account in the trusted group of user accounts for the given action;

monitoring, by the one or more network computing devices, inputs for the given action and about the item, the inputs about the item from the trusted group for the given action are used to classify the item as one of spam and non-spam, such that any input about the item from other than the trusted group of user accounts formed for the given action is excluded from being used to classify the item; and

evolving, by the one or more network computing devices, the trusted group based on modified trustworthiness scores of the plurality of user accounts for the given action, each modified trustworthiness score is determined, in part, by a comparison of an input about the item from a corresponding user account and inputs about the item from other user accounts in the plurality of user accounts, wherein evolving the trusted group further comprises at least one of moving at least one user account into the trusted group that previously was not in the trusted group and moving at least one user account out of the trusted group that was previously in the trusted group.

2. The method of claim 1, wherein the online activity features of the online user behavioral features comprising at least one of online browsing activity, electronic mail transactions and address book entries, and the user registration features of the static profile features comprising at least one of user preferences, age, gender and occupation.

3. The method of claim 1, wherein the given action is one of a plurality of actions, participation in more than one action of the plurality of actions results in a user account of the plurality of user accounts being assigned multiple trustworthiness scores, each trustworthiness score assigned to a given user account is associated with a different action of the plurality of actions, and wherein a different trusted group is identified for each action in the plurality of actions.

4. The method of claim 1, further comprising:

modifying the trustworthiness score assigned to each user account, comprising;

increasing a trustworthiness score of the given user account when input from a given user account matches the input from the other user accounts; and

decreasing the trustworthiness score of the given user account when input from a given user account does not match the input from the other user accounts.

5. The method of claim 1, further comprising:

modifying the trustworthiness score assigned to each user account, comprising;

increasing a trustworthiness score of the given user account when input from a given user account matches the input from the other user accounts and the input from the given user account matches a result from an anti-spam system; and

decreasing the trustworthiness score of the given user account when the input from the given user account is different than the input from other user accounts and the input from the given user account is different than the result from the anti-spam system.

6. The method of claim 1, further comprising:

when a trustworthiness score of a given user account is below a threshold value, identifying the given user account as a robot account; and

identifying the given user account as a human account when a trustworthiness score of the given user account is not below the threshold value.

7. The method of claim 1, further comprising:

modifying a spam filter based on the inputs from the trusted group.

8. A network device comprising:

a transceiver for communicating with at least one client device over a network;

a processor; and

a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic comprising;

identifying logic executed by the processor for identifying, for a given action, a trusted group of user accounts from a plurality of user accounts using a respective trustworthiness score for the given action assigned to each user account of the plurality, wherein the given action refers to an online activity that is performed by one or more users associated with respective user accounts, each user account'"'"'s trustworthiness score being determined using inputs received from the each user account for the given action, the trustworthiness score being used to identify the trusted group of user accounts whose input is to be used to classify an item and to identify other user accounts whose input is to be excluded from classifying the item, wherein the item refers to an article of the internet that can have an action performed on it, the identifying comprising generating an initial trustworthiness score for a user account of the plurality using a trained trustworthiness classifier and a feature set about the user account, the trained trustworthiness classifier comprising a number of machine-implemented algorithms used to evolve behaviors based on input data, the feature set comprising online user behavioral features and static profile features about the user account, wherein the online user behavioral features comprise online activity features and the static profile features comprise user registration features, the initial trustworthiness score for the user account is used at least initially to determine whether or not to include the user account in the trusted group of user accounts for the given action;

monitoring logic executed by the processor for monitoring inputs for the given action and about the item, the inputs about the item from the trusted group for the given action are used to classify the item as one of spam and non-spam, such that any input about the item from other than the trusted group of user accounts formed for the given action is excluded from being used to classify the item; and

evolving logic executed by the processor for evolving the trusted group based on modified trustworthiness scores of the plurality of user accounts for the given action, each modified trustworthiness score is determined, in part, by a comparison of an input about the item from a corresponding user account and inputs about the item from other user accounts in the plurality of user accounts, wherein evolving the trusted group further comprises at least one of moving at least one user account into the trusted group that previously was not in the trusted group and moving at least one user account out of the trusted group that was previously in the trusted group.

9. The network device of claim 8, wherein the given action is one of a plurality of actions, participation in more than one action of the plurality of actions results in a user account of the plurality of user accounts being assigned multiple trustworthiness scores, each trustworthiness score assigned to a given user account is associated with a different action of the plurality of actions, and wherein a different trusted group is identified for each action in the plurality of actions.

10. The network device of claim 8, wherein the program logic for execution by the processor further comprises:

modifying logic executed by the processor for modifying the trustworthiness score assigned to each user account, comprising;

increasing logic executed by the processor for increasing a trustworthiness score of the given user account when input from a given user account matches the input from the other user accounts; and

decreasing logic executed by the processor for decreasing the trustworthiness score of the given user account when the input from the given user account does not match the input from the other user accounts.

11. The network device of claim 8, wherein the program logic for execution by the processor further comprising:

identifying logic executed by the processor for identifying the given user account as a robot account when a trustworthiness score of a given user account is below a threshold value, identifying the given user account as a robot account; and

identifying logic executed by the processor for identifying the given user account as a human account when the trustworthiness score of the given user account is not below the threshold value.

12. The network device of claim 8, wherein the online activity features of the online user behavioral features comprising at least one of online browsing activity, electronic mail transactions and address book entries, and the user registration features of the static profile features comprising at least one of user preferences, age, gender and occupation.

13. A non-transitory computer-readable storage device having computer-executable instructions stored thereon such that when the storage device is accessed by a computing device, the instructions are executable by the computing device to perform actions, comprising:

identifying, for a given action, a trusted group of user accounts from a plurality of user accounts using a respective trustworthiness score for the given action assigned to each user account of the plurality, wherein the given action refers to an online activity that is performed by one or more users associated with respective user accounts, each user account'"'"'s trustworthiness score being determined using inputs received from the each user account for the given action, the trustworthiness score being used to identify the trusted group of user accounts whose input is to be used to classify an item, wherein the item refers to an article of the internet that can have an action performed on it, and to identify other user accounts whose input is to be excluded from classifying the item the identifying comprising generating an initial trustworthiness score for a user account of the plurality using a trained trustworthiness classifier and a feature set about the user account, the trained trustworthiness classifier comprising a number of machine-implemented algorithms used to evolve behaviors based on input data, the feature set comprising online user behavioral features and static profile features about the user account, wherein the online user behavioral features comprise online activity features and the static profile features comprise user registration features the initial trustworthiness score for the user account is used at least initially to determine whether or not to include the user account in the trusted group of user accounts for the given action;

monitoring inputs for the given action and about an item, the inputs about the item from the one or more trusted groups for the given action are used to classify the item as one of spam and non-spam, such that any input about the item from other than the trusted group of user accounts formed for the given action is excluded from being used to classify the item; and

evolving the trusted group based on modified trustworthiness scores of the plurality of user accounts for the given action, each modified trustworthiness score is determined, in part, by a comparison of an input about the item from a corresponding user account and inputs about the item from other user accounts in the plurality of user accounts, wherein evolving the trusted group further comprises at least one of moving at least one user account into the trusted group that previously was not in the trusted group and moving at least one user account out of the trusted group that was previously in the trusted group.

14. The non-transitory computer-readable storage device of claim 13, wherein the online activity features of the online user behavioral features comprising at least one of online browsing activity, electronic mail transactions and address book entries, and the user registration features of the static profile features comprising at least one of user preferences, age, gender and occupation.

15. The non-transitory computer-readable storage device of claim 13, wherein the given action is one of a plurality of actions, participation in more than one action of the plurality of actions results in a user account of the plurality of user accounts being assigned multiple trustworthiness scores, each trustworthiness score assigned to a given user account is associated with a different action of the plurality of actions, and wherein a different trusted group is identified for each action in the plurality of actions.

16. The non-transitory computer-readable storage device of claim 13, wherein the one or more other network devices further enables actions, the actions comprising:

modifying the trustworthiness score assigned to each user account, comprising;

when input from a given user account matches the input from the other user accounts, increasing a trustworthiness score of the given user account; and

decreasing the trustworthiness score of the given user account when input from the given user account does not match the input from the other user accounts.

17. The non-transitory computer-readable storage device of claim 13, wherein the one or more other network devices further enables actions, the actions comprising:

when a trustworthiness score of a given user account is below a threshold value, identifying the given user account as a robot account; and

identifying the given user account as a human account when the trustworthiness score of the given user account is not below the threshold value.