DETECTING ONLINE ABUSE IN IMAGES

• US 20080008348A1
• Filed: 02/01/2007
• Published: 01/10/2008
• Est. Priority Date: 02/01/2006
• Status: Active Grant

• Current Assignee: Opsec Online Limited
• Original Assignee: MarkMonitor Inc.
• Inventors: Shull, Mark, Metois, Eric, Wolosewicz, Jack
• Granted Patent: US 7,925,044 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/18 Extraction of features or characteristics of the image

Abstract

Tools for analyzing images are disclosed. In some cases, the images are analyzed in order to determine whether a particular web site and/or email message is part of an illegitimate online activity. In an aspect, an image analysis process comprises comparing a suspect image with one or more elements of interest (which can include images, words, etc.) by generating fingerprints characterizing the suspect image and the elements of interest, to allow for a quantitative comparison.

Claims

1. A method of monitoring a web site, the method comprising:

maintaining a database comprising representations of a plurality of elements of interest;

downloading a suspect image from the web site;

preprocessing the suspect image with a computer to identify at least one suspect blob in the image, the suspect blob having an associated set of suspect blob characteristics and an associated set of suspect blob boundaries;

based at least in part of the set of suspect blob boundaries, creating, with the computer, a suspect fingerprint for the suspect blob, the suspect fingerprint comprising a set of suspect fingerprint signals;

based at least in part on the set of blob characteristics, identifying, from a database of image templates, at least one candidate fingerprint comprising a plurality of candidate fingerprint signals, the at least one candidate fingerprint being associated with an element of interest;

calculating a plurality of similarity measures between the suspect fingerprint and the candidate fingerprint, each of the similarity measures being calculated based on a suspect fingerprint signal a respective candidate fingerprint signal;

weighting each of the plurality of similarity measures;

calculating a composite similarity score for the candidate blob, based on the weighted similarity measures;

identifying a minimum individual similarity score from among the plurality of similarity measures;

comparing the composite similarity score with a first threshold;

comparing the minimum individual similarity score with a second threshold;

based at least in part on a comparison of the composite similarity score with the first threshold and a comparison of the minimum similarity score with the second threshold, determining whether the suspect image is a likely match for the element of interest; and

determining whether the web site is likely to be part of an illegitimate online activity, based at least in part on a determination of whether the suspect image is a likely match for the element of interest.

2. A method as recited in claim 1, further comprising notifying a user if it is determined that the web site is likely to be part of an illegitimate online activity.

3. A method as recited in claim 1, wherein each of the plurality of fingerprint signals is selected from the groups consisting of an X projection signal, a Y mean signal, a Y standard deviation signal, a transitions signal, a top edge signal, a bottom edge signal, and a binary template signal.

4. A method as recited in claim 1, wherein preprocessing the suspect image comprises:

converting the image to a grayscale image;

passing the grayscale image through a Gaussian low pass filter to produce a filtered image;

determining a set of tonal gradients of the filtered image;

defining a set of edge boundaries in the image, based at least on part on the set of tonal gradients;

identifying one or more blobs in the filtered image, based at least in part on the set of edge boundaries;

generating a blobmap for the filtered image, based at least in part on the identified one or more blobs;

defining a set of rectangular boundaries for each of the identified blobs, based at least in part on the blobmap; and

deriving a set of blob characteristics for each of the identified blobs, wherein a set of blob characteristics comprises an aspect ratio of a blob and a normalized area of a blob.

5. A method as recited in claim 4, wherein determining a set of tonal gradients comprises:

computing a horizontal tonal derivative of the filtered image; and

computing a vertical tonal derivative of the filtered image.

6. A method as recited in claim 4, wherein generating a blobmap comprises applying a region growing stage of each of the identified blobs to join one or more pixels in two different blobs that should belong to the same blob.

7. A method as recited in claim 4, wherein preprocessing the suspect image further comprises applying a blob merging function to merge two or more blobs that have overlapping boundaries.

8. A method as recited in claim 1, wherein creating a suspect blob fingerprint comprises calculating a set of fingerprint signals, the set of fingerprint signals comprising:

a first fingerprint signal corresponding to an X projection of the blob;

a second fingerprint signal corresponding to a mean Y value of the blob;

a third fingerprint signal corresponding to a standard deviation of a Y value of the blob;

a fourth fingerprint signal corresponding to a top edge of the blob;

a fifth fingerprint signal corresponding to a bottom edge of the blob;

a sixth fingerprint signal corresponding to a set of transitions within the blob; and

a seventh fingerprint signal corresponding to a binary template of the blob.

9. A method as recited in claim 8, wherein generating a suspect blob fingerprint further comprises applying a signal conditioning function to at least one fingerprint signal in the set of fingerprint signals.

10. A method of monitoring a web site, the method comprising:

maintaining a database comprising representations of a plurality of elements of interest;

downloading a suspect image from the web site;

analyzing the suspect image to determine whether the suspect image matches any of the elements of interest; and

determining whether the web site is likely to be part of an illegitimate online activity, based at least in part upon an analysis of the suspect image.

11. A method as recited in claim 10, wherein analyzing the image comprises:

segmenting the suspect image into a plurality of blobs, each of the plurality of blobs being described by a set of characteristics;

for each of the plurality of blobs, creating a fingerprint for the blob, based at least on the characteristics of the blob;

for at least one of the plurality of blobs, collecting from the database a list of one or more candidate elements, based on the characteristics the at least one of the plurality of blobs;

for each of the one or more candidate elements, assigning a score to the candidate element based on similarities between the representation of the candidate element and the fingerprint of the at least one of the plurality of blobs; and

based at least in part on the scores assigned to the candidate elements, determining whether the image is a likely match for one or more of the candidate elements.

12. A method as recited in claim 11, wherein the set of characteristics describing a blob comprises one or more characteristics selected from the group consisting of:

an aspect ratio and a normalized area.

13. A method of monitoring email messages, the method comprising:

maintaining a database comprising representations of a plurality of elements of interest;

receiving an email message;

analyzing the email message to identify a suspect image in the email message;

analyzing the suspect image to determine whether the suspect image matches any of the elements of interest; and

determining whether the email message is likely to be part of an illegitimate online activity, based at least in part upon an analysis of the suspect image.

14. A method as recited in claim 13, wherein analyzing the suspect image comprises:

preprocessing the suspect image with a computer to identify at least one suspect blob in the image, the suspect blob having an associated set of suspect blob characteristics and an associated set of suspect blob boundaries;

based at least in part of the set of suspect blob boundaries, creating, with the computer, a suspect blob fingerprint for the suspect blob, the suspect blob fingerprint comprising a set of suspect blob signals;

based at least in part on the set of blob characteristics, identifying, from a database of image templates, at least one candidate fingerprint comprising a plurality of candidate fingerprint signals, the at least one candidate fingerprint being associated with an element of interest;

calculating a plurality of similarity measures between the suspect fingerprint and the candidate fingerprint, each of the similarity measures being based on a suspect blob signal or characteristic and a respective candidate blob signal or characteristic;

weighting each of the plurality of similarity measures;

calculating a composite similarity score for the candidate blob, based on the weighted similarity measures;

identifying a minimum individual similarity score from among the plurality of similarity measures;

comparing the composite similarity score with a first threshold;

comparing the minimum individual similarity score with a second threshold; and

based at least in part on a comparison of the composite similarity score with the first threshold and a comparison of the minimum similarity score with the second threshold, determining whether the suspect image is a likely match for the element of interest.

15. A method of analyzing an image, the method comprising:

maintaining a database comprising representations of a plurality of elements of interest;

segmenting the image into a plurality of blobs, each of the plurality of blobs being described by a set of characteristics and having a set of boundaries;

creating, for each of the plurality of blobs, a fingerprint for the blob, based at least in part on the boundaries of the blob;

collecting, for at least one of the plurality of blobs and from the database, a list of one or more candidate elements, based on the characteristics the at least one of the plurality of blobs;

assigning, for each of the one or more candidate elements, a score to the candidate element based on similarities between a fingerprint of the candidate element and the fingerprint of the at least one of the plurality of blobs; and

determining, based at least in part on the scores assigned to the candidate elements, whether the image is a likely match for one or more of the candidate elements.

16. A method as recited in claim 15, wherein a representation of an element of interest comprises an image template for the element of interest, the image template comprising a fingerprint for the element of interest, the fingerprint comprising a plurality of fingerprint signals.

17. A method as recited in claim 15, further comprising providing feedback about whether the image is a likely match for one or more of the candidate elements.

18. A method as recited by claim 17, wherein providing feedback comprises displaying for a user one or more candidate elements that are determined to be likely matches for the image.

19. A system comprising:

a processor;

a database comprising representations of a plurality of elements of interest; and

a set of instructions executable by the processor, the set of instructions comprising;

a) instructions to maintain a database comprising representations of a plurality of elements of interest;

b) instructions to segment the image into a plurality of blobs, each of the plurality of blobs being described by a set of characteristics and having a set of boundaries;

c) instructions to create, for each of the plurality of blobs, a fingerprint for the blob, based at least in part on the boundaries of the blob;

d) instructions to collect, for at least one of the plurality of blobs and from the database, a list of one or more candidate elements, based on the characteristics the at least one of the plurality of blobs;

e) instructions to assign, for each of the one or more candidate elements, a score to the candidate element based on similarities between a fingerprint of the candidate element and the fingerprint of the at least one of the plurality of blobs; and

f) instructions to determine, based at least in part on the scores assigned to the candidate elements, whether the blob is a likely match for one or more of the candidate elements.

20. A software program comprising a set of instructions executable by one or more processors, the set of instructions comprising:

instructions to maintain a database comprising representations of a plurality of elements of interest;

instructions to segment the image into a plurality of blobs, each of the plurality of blobs being described by a set of characteristics and having a set of boundaries;

instructions to create, for each of the plurality of blobs, a fingerprint for the blob, based at least in part on the boundaries of the blob;

instructions to collect, for at least one of the plurality of blobs and from the database, a list of one or more candidate elements, based on the characteristics the at least one of the plurality of blobs;

instructions to assign, for each of the one or more candidate elements, a score to the candidate element based on similarities between a fingerprint of the candidate element and the fingerprint of the at least one of the plurality of blobs; and

instructions to determine, based at least in part on the scores assigned to the candidate elements, whether the blob is a likely match for one or more of the candidate elements.

21. A computer system for analyzing an image, the system comprising:

a database configured to store of image templates for a plurality of elements of interest, each of the image templates comprising a candidate fingerprint, each candidate fingerprint comprising a plurality of candidate fingerprint signals;

a processor in communication with the database; and

a computer readable medium having embodied thereon a software program executable by the processor, the software program comprising;

a) an image preprocessing and segmentation component configured to receive a suspect image and identify, within the suspect image, a suspect blob having an associated set of suspect blob boundaries;

b) a fingerprint generation component configured to create a suspect fingerprint, based at least in part on the suspected blob boundaries, the suspect blob fingerprint comprising a set of suspect blob fingerprint signals; and

c) a fingerprint matching component configured to calculate a similarity score for an element of interest, the similarity score being based on a plurality of similarity measures between the suspect fingerprint and a candidate fingerprint for the element of interest, each of the similarity measures being calculated based on a suspect fingerprint signal and a respective candidate fingerprint signal, and to determine, based at least in part on the similarity score, determine whether the suspect image is a likely match for the element of interest.

22. A computer system as recited in claim 21, wherein the fingerprint matching component is configured to calculate similarity scores for a plurality of elements of interest, and based on the respective similarity scores, determine which of the elements of interest is a best match for the suspect image.

23. A computer system as recited in claim 21, wherein the fingerprint matching component is configured to calculate a plurality of similarity scores, the plurality of similarity scores comprising:

a composite similarity score based on a weighted composite of the plurality of similarity measures; and

a minimum individual similarity score based on a minimum value from among the plurality of similarity measures.

24. A computer system as recited in claim 23, wherein the determining whether the suspect image is a likely match comprises:

comparing the composite similarity score with a first threshold having a first value; and

comparing the minimum individual similarity score with a second threshold having a second value.

25. A computer system as recited in claim 24, wherein the first value and the second value are the same value.