URL Reputation System

• US 20110191342A1
• Filed: 02/01/2010
• Published: 08/04/2011
• Est. Priority Date: 02/01/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lin, Yun, Arai, Benjamin, de Oliveira, Jose Marcos, Boucher, Craig, Waggoner, Nicholas, COHEN, Jason
• Granted Patent: US 8,229,930 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A URL reputation system may have a reputation server and a client device with a cache of reputation information. A URL reputation query from the client to the server may return reputation data along with probabilistic set membership information for several variants of the requested URL. The client may use the probabilistic set membership information to determine if the reputation server has additional information for another related URL as well as whether the classifications are inheritable from one of the variants. If the probabilistic set membership determines that the reputation server may have additional information, a query may be made to the reputation server, otherwise the reputation may be inferred from the data stored in the cache.

Claims

1. A method performed by a URL reputation client, said method being performed on a computer processor, said method comprising:

receiving a first URL;

determining that said first URL is not present in a cached reputation database;

determining a set of variants for said first URL;

determining that a first variant is stored within said cached reputation database, said first variant having a first classification;

determining that said first classification is inheritable;

analyzing said cached reputation database to determine that a reputation server does not contain classification information about said first URL, said cached reputation database comprising probabilistic set membership keys; and

assigning said first classification to said first URL.

2. The method of claim 1 further comprising:

identifying a second URL;

analyzing said cached reputation database to determine that said second URL is included in one of said probabilistic set membership keys;

transmitting said second URL to a reputation server; and

receiving a second classification, said second classification being defined for said second URL.

3. The method of claim 2 further comprising:

for each of a plurality of URL variants for said second URL, receiving a probabilistic set membership key; and

storing said probabilistic set membership keys in said cached reputation database.

4. The method of claim 3, each of said probabilistic set membership keys having an expiration setting.

5. The method of claim 3 further comprising:

determining said plurality of URL variants; and

transmitting said plurality of URL variants to said reputation server.

6. The method of claim 1, said first classification comprising a first inheritance indicator.

7. The method of claim 6, said first inheritance indicator indicating that said reputation classification is inheritable by path variant.

8. The method of claim 6, said first inheritance indicator indicating that said reputation classification is inheritable by host variant.

9. The method of claim 1, said first classification comprising membership in a first class and a second class.

10. The method of claim 9, said first class having a first inheritance indicator and said second class having a second inheritance indicator.

11. The method of claim 10, said first class being inheritable by path variant, said second class being inheritable by host variant.

12. A URL reputation server comprising:

a processor;

a network connection;

a URL reputation database;

said processor configured to;

receive a URL as part of a request for a URL reputation, said request being sent from a client device;

determine said URL reputation and a probabilistic set membership key for said URL; and

transmit said URL reputation and said probabilistic set membership key to said client device.

13. The URL reputation server of claim 12 further comprising:

a web crawler; and

a classification engine configured to determine a classification for a given URL and populate said URL reputation database with said classification.

14. The URL reputation server of claim 12, said processor further configured to:

determine a plurality of variants for said URL;

determine a probabilistic set membership key for each of said plurality of variants; and

transmit said probabilistic set membership keys to said client device in response to said request.

15. The URL reputation server of claim 12, said probabilistic set membership key being determined for a single generation of addresses.

16. The URL reputation server of claim 12, said probabilistic set membership key being determined for a plurality of generations of addresses.

17. The URL reputation server of claim 12, said probabilistic set membership key having an expiration.

18. A URL reputation client comprising:

a processor;

a network connection;

a cached URL reputation database comprising reputation information for a plurality of URL addresses having a common second level domain name, and for each of said plurality of URL addresses, a probabilistic set membership key for lower level URL addresses, said probabilistic set membership key indicating availability of said lower level URL addresses in a URL reputation database available from a URL reputation server;

said processor configured to;

receive a first URL;

determine that said first URL is not present in a cached URL reputation database;

determine a first set of variants comprising variants for said first URL;

determine that a first variant is stored within said cached reputation database, said first variant having a first classification;

determine that said first classification is inheritable;

analyze said cached reputation database to determine that said reputation server does not contain classification information about said first URL, said cached reputation database comprising probabilistic set membership keys; and

assign said first classification to said first URL.

19. The URL reputation client of claim 18, said processor further configured to:

receive a second URL;

determine that said second URL is not present in a cached URL reputation database;

determine a second set of variants comprising variants for said second URL;

determine that none of said second set of variants are stored within said cached reputation database;

transmitting said second URL to said URL reputation server; and

receiving a second classification assigned to said second URL from said URL reputation server.

20. The URL reputation client of claim 19, said processor further configured to:

transmit said second set of variants to said URL reputation server.