System and method of identifying advertisement in images

• US 8,600,102 B1
• Filed: 09/19/2011
• Issued: 12/03/2013
• Est. Priority Date: 09/19/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Frome, Andrea
• Primary Examiner(s): Wang, Claire X
• Assistant Examiner(s): JAMES, THOMAS AIDEN
• Application Number:
  US13/236,032
• Time in Patent Office: 806 Days
• Field of Search: None
• US Class Current: 382/100
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G09F 9/30 in which the desired character or characters are formed by combining individual elements panels comprising a number of electrodes in a single cell controlling light arriving from an independent light source, e.g. electro-optical or magneto-optical cell, G02F1/00
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

A system and method is provided wherein, in one aspect, a processor determines whether multiple street level images have captured a nearly-identical face. If so, the images are processed to determine whether the face appears to be part of an advertisement. Once it is determined that the face is displayed on an advertisement, the boundaries of the advertisement may be determined and the location of the advertisement is stored for future use, e.g., potentially replacing the advertisement in the image with a different advertisement.

Claims

1. A method of determining a presence of an advertisement in an image, the method comprising:

receiving an image related to a first geographic location;

detecting, with a processor, a face in the image;

comparing, with the processor, the detected face to a plurality of faces found in a plurality of images, at least one of the plurality of images being related to a second geographic location;

determining, based on the comparison, whether an advertisement with the detected face is present in the received image; and

analyzing, with the processor, image features outside the detected face when it is determined that an advertisement is present in the received image, the processor.

2. The method of claim 1, wherein detecting a face in the image comprises analyzing the image with a sliding-window face detector tuned based on recall and precision.

3. The method of claim 1, wherein comparing the detected face to the plurality of faces comprises:

fingerprinting the detected face to represent the detected face in a data set of a vector of numbers; and

comparing the fingerprint of the detected face to fingerprints for corresponding ones of the plurality of faces.

4. The method of claim 3, wherein the fingerprinting is performed using Haar wavelet decomposition.

5. The method of claim 3, wherein the fingerprinting is performed with SIFT or Congas features of the detected face.

6. The method of claim 3, wherein the comparing further comprises hashing the vector of numbers through locality sensitive hashing or standard string hashing.

7. The method of claim 1, wherein determining if the advertisement is present comprises finding, from the plurality of faces, at least one face identical to the detected face.

8. The method of claim 1, wherein determining if the advertisement is present comprises finding, from the plurality of faces, at least one face where the similarity between the detected face and the at least one face is above a predetermined threshold.

9. The method of claim 1, wherein analyzing the image features comprises:

extracting image features of areas outside the face in the received image; and

matching the extracted image features to image features of areas outside the plurality of faces in the plurality of images.

10. The method of claim 1, wherein analyzing the image features comprises determining a boundary of the advertisement.

11. The method of claim 10, wherein the boundary of the advertisement is determined through edge detection.

12. The method of claim 1, wherein at least another one of the plurality of images is related to the first geographic location and was captured at a time different from a time where the received image was taken.

13. The method of claim 1, wherein the advertisement comprises one of a billboard, a poster, and a banner.

14. A system of determining presence of an advertisement in an image, the system comprising:

a processor; and

a memory accessible to the processor, the memory storing instructions executable by the processor, the instructions comprising instructions for;

receiving an image related to a first geographic location;

detecting, with a processor, a face in the image;

comparing, with the processor, the detected face to a plurality of faces found in a plurality of images, at least one of the plurality of images being related to a second geographic location;

determining, based on the comparison, whether an advertisement with the detected face is present in the received image; and

analyzing image features outside the detected face when it is determined that an advertisement is present in the received image.

15. The system of claim 14, wherein detecting a face in the image comprises analyzing the image with a sliding-window face detector tuned based on recall and precision.

16. The system of claim 14, wherein the instructions for comparing comprises instructions for:

fingerprinting the detected face to represent the detected face in a data set of a vector of numbers; and

comparing the fingerprint of the detected face to fingerprints for corresponding ones of the plurality of faces.

17. The system of claim 16, wherein the fingerprinting is performed using Haar wavelet decomposition.

18. The system of claim 16, wherein the fingerprinting is performed with SIFT or Congas features of the detected face.

19. The system of claim 16, wherein the instructions for comparing further comprises instructions for hashing the vector of numbers through locality sensitive hashing or standard string hashing.

20. The system of claim 14, wherein the instructions for determining if the advertisement is present comprises instructions for finding, from the plurality of faces, at least one face identical to the detected face.

21. The system of claim 14, wherein the instructions for determining if the advertisement is present comprises instructions for finding from the plurality of faces at least one face where the similarity between the detected face and the at least one face is above a predetermined threshold.

22. The system of claim 14, wherein the instructions for analyzing the image features comprises instructions for:

extracting image features of areas outside the face in the received image; and

matching the extracted image features to image features of areas outside the plurality of faces in the plurality of images.

23. The system of claim 14, wherein the instructions for analyzing the image features comprises instructions for determining a boundary of the advertisement.

24. The system of claim 23, wherein the boundary of the advertisement is determined through edge detection.

25. The system of claim 14, wherein at least another one of the plurality of images is related to the first geographic location and was captured at a time different from the time the image was taken.

26. The system of claim 14, wherein the advertisement comprises a billboard, a poster, and a banner.

27. A method of analyzing an image comprising:

selecting a first and second street level image from among a plurality of street level images, wherein the street level images were captured at first and second times and first and second locations;

detecting, with a processor, whether the first and second street level image contain an image associated with a face;

for each face image detected in the first street level image, determining whether the face image is similar to a face image detected in the second image;

determining, with a processor, whether the first level image has captured an advertisement based on whether the similarity of the detected face images from the first and second images indicate that the faces captured in the street level images are likely to be identical, and further based on the location and time that the street level images were captured; and

identifying, with a processor, a portion of the first street level image that is likely to have captured an advertisement based on the image information surrounding the detected face image of the first street level image.

28. The method of claim 27 wherein determining whether each street level image has captured an advertisement based on the location and time that the street level images were captured comprises determining whether the time and location of the first and second street level image are sufficiently different to make it unlikely that the detected face images are of the same person being physically present at the first and second locations at the time the street level images were captured.

29. The method of claim 28 wherein the first and second locations are the same and the differences in capture times make it unlikely that the detected face images are of the same person being physically present at the first and second locations at the time the street level images were captured.

30. The method of claim 28 comprising displaying the first street level image to a user with a new advertisement replacing the advertisement captured in the street level image.

31. The method of claim 27 wherein identifying a portion of the first street level image that is likely to reflect an advertisement comprises using edge detection to identify the boundaries of an advertisement containing the detected face image.

32. The method of claim 27 further comprising detecting, with a processor, whether a third street level image contains an image associated with a face, and wherein determining, with a processor, whether the first street level image has captured an advertisement is further based on whether the similarity of the detected face images from the first, second and third images indicate that the faces captured in the three street level images are likely to be identical, and further based on the location and time that the street level images were captured.

33. The method of claim 27 wherein determining whether each street level image has captured an advertisement further comprises analyzing the three-dimensional location of the surface of the face.

34. The method of claim 27 wherein determining whether each street level image has captured an advertisement further comprises analyzing the three-dimensional location of the surfaces surrounding the face.