Image search privacy protection techniques

• US 9,098,584 B1
• Filed: 07/19/2012
• Issued: 08/04/2015
• Est. Priority Date: 07/19/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fredinburg, Dan, Graber, Ross, Swerdlow, Andrew, Klein, Peter
• Primary Examiner(s): Liao, Jason
• Assistant Examiner(s): Mitiku, Berhanu
• Application Number:
  US13/553,500
• Time in Patent Office: 1,111 Days
• Field of Search: 382/115, 382/118, 705/7.34, 705/14.57, 705/14.58, 705/26.9, 707/713, 707723-724, 707/758, 707/769, 707/722, 707/728, 707/E17.019, 707/E17.059, 707/732, 707/754
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2458 Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

A unified search entity, which provides at least visual query based web searching and an associated mechanism for expressing privacy preferences, obtains, from a requester, a visual query including at least a first facial image. The unified search entity identifies, via facial recognition on the at least first facial image, an identity of an individual subject of the at least first facial image and/or at least a second facial image of the individual subject associated with the at least first facial image. The unified search entity obtains, from the individual, via the mechanism for expressing privacy preferences, a plurality of privacy preferences including a visual query preference. The identity of the individual subject and/or the at least second facial image are communicated from the unified search entity to the requester only if permitted by the visual query preference.

Claims

1. A method of controlling access to visual query results, the method comprising the steps of:

obtaining, by a unified search entity, from a requester, a visual query comprising at least a first facial image, the unified search entity comprising and entity the providers at least visual query based web searching and an associated mechanism for expressing privacy preference that provides for individual subject of the visual query to express the privacy preferences to suppress the requester from receiving a visual query result that is associated with the individual, for the requester not being logged into an online account of the unified search entity, or the requester not being in a social group of the individual;

Identifying, by the unified search entity, via facial recognition on the at least first facial image, of at least one of;

an identity of an individual subject of the at least first facial image; and

at least a second facial image associated with the at least first facial image, the at least second facial image being of the individual subject;

obtaining, by the unified search entity, from the individual subject of the least first facial image, via the mechanism for expressing privacy preference, a plurality of privacy preferences including a visual query preference of the individual subject of the at least first facial image; and

for the visual query preference of the individual subject permitting the requester to receive the visual query result, communicating at least one of the identity of the individual subject and the at least second facial image, from the unified search entity, to the requester; and

for the visual query preference of the individual subject suppressing the requester from receiving the visual query result, providing the requester with one of, an indication that the individual exists but does not permit the visual query of his or her facial image by a user not in the social group of the individual, and an option to invite the individual to a sharing group.

2. A method of controlling access to visual query results, the method comprising the steps of:

obtaining, by a search entity, from a requester, a visual query comprising at least a first facial image;

identifying, by the search entity, via facial recognition on the at least first facial image, general web content, from web pages not under control of the search entity, the general web content being associated with an individual subject of the first facial image;

obtaining, by the search entity, from the individual subject of the at least first facial image, via a mechanism for expressing privacy preferences, at least one privacy preference comprising a visual query preference of the individual subject of the at least first facial image, wherein the mechanism provides for an individual subject of the visual query to express the privacy preferences to suppress the requester from receiving a visual query result that is associated with the individual, for the requester not being logged into an online account of the unified search entity, or the requester not being in a social group of the individual; and

for the visual query preference of the individual subject permitting the requester to receive the visual query result, communicating at least a portion of the general web content, from the search entity, to the requester; and

for the visual query preference of the individual subject suppressing the requester from receiving the visual query result, providing the requester with one of an indication that the individual exists but does not permit the visual query of his or her facial image by a user not in the social group of the individual, and an option to invite the individual to a sharing group.

3. The method of claim 2, wherein the identifying comprises:

identifying the individual subject by performing the facial recognition to compare the at least first facial image to a database of stored images linked to corresponding people, the individual subject being one of the corresponding people; and

identifying the general web content by carrying out a term-based general web search on the individual subject'"'"'s name.

4. The method of claim 3, wherein the database of stored images is under control of the search entity.

5. The method of claim 2, wherein the identifying comprises:

performing the facial recognition to compare the at least first facial image to an index database of general web content, obtained by a crawler, the index database of general web content including facial image data for comparison with the at least first facial image during the facial recognition.

6. The method of claim 2, further comprising determining, by the search entity, whether the requester has logged in;

wherein the visual query preference specifies different treatment for the communicating of the general web content, from the search entity, to the requester, depending on whether the requester has logged in.

7. A method of alerting an entity to use of an image, the method comprising the steps of:

obtaining access to a first database having a plurality of first database records, each of the first database records comprising at least;

an identification of a corresponding one of a plurality of entities; and

a plurality of features of a plurality of images associated with the entities;

obtaining access to a second database having a plurality of second database records, each of the second database records comprising at least;

an identification of a corresponding one of a plurality of generalized web locations; and

a plurality of features of a plurality of images associated with the generalized web locations, the plurality of features being extracted from images obtained by crawling the generalized web locations;

for at least one of the first database records, comparing corresponding ones of the features of a corresponding one of the images associated with a corresponding one of the entities to the plurality of features in the second database records to determine an identified entity of the plurality of entities;

obtaining, from the identified entity, via a mechanism for expressing privacy preferences, at least one privacy preference comprising a visual query preference of the identified entity, wherein the mechanism provides for the identified entity of the comparing to express the privacy preferences to suppress a requester of the comparing from the receiving a visual query result that is associated with the identified entity, for the requester not being logged into an online account, or the requester not being in a social group of the identified entity, and further, the mechanism allows the individual subject of the query to express a preference for receiving an alert indicative of a use of the facial image of the identified entity in the second database records, and an online delivery address and frequency preference for the comparing and for receiving the alert; and

advising the identified entity, corresponding to the at least one of the first database records, if the comparing yields a probable match.

8. The method of claim 7, wherein the advising comprises sending the given one of the entities an e-mail alert.

9. The method of claim 8, further comprising obtaining from the given one of the entities a selection of the e-mail alert during a registration process.

10. The method of claim 7, wherein the advising comprises displaying a notification to the given one of the entities when the given one of the entities accesses a unified privacy dashboard.

11. The method of claim 7, further comprising displaying to the given one of the entities a given one of the plurality of images associated with a given one of the generalized web locations corresponding to matching ones of the features in the second database records.

12. The method of claim 7, further comprising affording the identified entity an opportunity to remediate improper use by the requester a given one of the plurality of images associated with a given one of the generalized web locations corresponding to matching ones of the features in the second database records, wherein the remediating of the improper user comprises submitting a request by the identified entity for the improper use of the given one of the images to be removed from the second database.

13. The method of claim 7, wherein:

in the step of obtaining access to the first database, the entities comprise a first group of humans and the plurality of images associated with the entities comprise photographs of the first group of humans;

in the step of obtaining access to the second database, the plurality of images associated with the web locations comprise photographs of a second group of humans; and

the comparing step comprises performing facial recognition.

14. The method of claim 13, wherein, in the step of obtaining access to the second database, the photographs of the second group of humans comprise still photographs.

15. The method of claim 13, wherein, in the step of obtaining access to the second database, the photographs of the second group of humans comprise frames of videos.

16. The method of claim 7, wherein:

in the step of obtaining access to the first database, the entities comprise enterprises and the plurality of images associated with the entities comprises logos of the enterprises;

in the step of obtaining access to the second database, the plurality of images associated with the web locations comprise logos of enterprises; and

the comparing step comprises performing pattern recognition.

17. The method of claim 7, further comprising:

crawling the generalized web locations to obtain the plurality of images associated with the generalized web locations; and

extracting the plurality of features of the plurality of images associated with the generalized web locations, to populate the second database.

18. The method of claim 7, further comprising populating the first database by obtaining tags of images at a social network server.

19. The method of claim 7, wherein:

in the step of obtaining access to the first database, the pluralities of features of the plurality of images associated with the entities are stored as first database feature vectors; and

in the step of obtaining access to the second database, the plurality of features of the plurality of images associated with the generalized web locations are stored as second database feature vectors.

20. The method of claim 7, further comprising providing a system, wherein the system comprises distinct software modules, each of the distinct software modules being embodied in a non-transitory manner on a tangible computer-readable recordable storage medium, and wherein the distinct software modules comprise a first database module, a second database module, and a search system module;

wherein;

the step of obtaining access to the first database is carried out by the first database module executing on at least one hardware processor;

the step of obtaining access to the second database is carried out by the second database module executing on at least one hardware processor;

the step of comparing is carried out by the search system module executing on at least one hardware processor; and

the step of advising the given one of the entities is initiated by the search system module executing on at least one hardware processor.

21. A method of alerting a person to appearance in a video, the method comprising the steps of:

obtaining access to a first database having a plurality of first database records, each of the first database records comprising;

an identification of a corresponding one of a plurality of humans; and

features of a corresponding one of a plurality of photographs associated with the corresponding one of the plurality of humans;

deconstructing a plurality of videos into individual frames;

extracting features associated with images of human faces contained in the individual frames;

populating a second database with a plurality of second database records, each of the second database records comprising;

an identifier of a corresponding one of the videos; and

corresponding ones of the features associated with the images of the human faces contained in the corresponding one of the videos;

for at least one of the first database records, comparing the features of corresponding one of a plurality of photographs associated with the corresponding one of the plurality of humans to the features in the second database records to determine an identified human of the plurality of humans;

obtaining, from the identified human, via a mechanism for expressing privacy preferences, at least one privacy preference comprising a visual query preference of the identified entity, wherein the mechanism provides for the identified entity of the comparing to express the privacy preferences to suppress a requester of the comparing from the receiving a visual query result that is associated with the identified entity, for the requester not being logged into an online account, or the requester not being in a social group of the identified entity, and further, the mechanism allows the individual subject of the query to express a preference for receiving an alert indicative of a use of the facial image of the identified entity in the second database records, and an online delivery address and frequency preference for the comparing and for receiving the alert; and

advising the identified entity, corresponding to the at least one of the first database records, if the comparing yields a probable match.

22. The method of claim 21, wherein:

in the step of obtaining access to the first database, the features of the corresponding one of the plurality of photographs associated with the corresponding one of the plurality of humans are stored as first database feature vectors; and

in the step of populating the second database, the corresponding ones of the features associated with the images of the human faces contained in the corresponding one of the videos are stored as second database feature vectors.

23. An article of manufacture comprising a computer program product for alerting an entity to use of an image, the computer program product comprising:

a tangible computer-readable recordable storage medium, storing a non-transitory manner computer readable program code, the computer readable program code comprising;

computer readable program code configured to obtain access to a first database having a plurality of first database records, each of the first database records comprising at least;

an identification of a corresponding one of a plurality of entities; and

a plurality of features of a plurality of images associated with the entities;

computer readable program code configured to obtain access to a second database having a plurality of second database records, each of the second database records comprising at least;

an identification of a corresponding one of a plurality of generalized web locations; and

a plurality of features of a plurality of images associated with the generalized web locations, the plurality of features being extracted from images obtained by crawling the generalized web locations;

computer readable program code configured to, for at least one of the first database records, compare corresponding ones of the features of a corresponding one of the images associated with a corresponding one of the entities to the plurality of features in the second database records to determine an identified entity of the plurality of entities;

computer readable program code configured to obtain, from the identified entity, via a mechanism for expressing privacy preferences, at least one privacy preference comprising a visual query preference of the identified entity, wherein the mechanism provides for the identified entity of the comparing to express the privacy preferences to suppress a requester of the comparing from the receiving a visual query result that is associated with the identified entity, for the requester not being logged into an online account, or the requester not being in a social group of the identified entity, and further, the mechanism allows the individual subject of the query to express a preference for receiving an alert indicative of a use of the facial image of the identified entity in the second database records, and an online delivery address and frequency preference for the comparing and for receiving the alert; and

computer readable program code configured to initiate advising the identified entity, corresponding to the at least one of the first database records, if the comparing yields a probable match.

24. An apparatus for alerting an entity to use of an image, the apparatus comprising:

a memory; and

at least one processor, coupled to the memory, and operative to;

obtain access to a first database having a plurality of first database records, each of the first database records comprising at least;

an identification of a corresponding one of a plurality of entities; and

a plurality of features of a plurality of images associated with the entities;

obtain access to a second database having a plurality of second database records, each of the second database records comprising at least;

an identification of a corresponding one of a plurality of generalized web locations; and

a plurality of features of a plurality of images associated with the generalized web locations, the plurality of features being extracted from images obtained by crawling the generalized web locations;

for at least one of the first database records, compare corresponding ones of the features of a corresponding one of the images associated with a corresponding one of the entities to the plurality of features in the second database records to determine an identified entity of the plurality of entities;

obtain, from the identified entity, via a mechanism for expressing privacy preferences, at least one privacy preference comprising a visual query preference of the identified entity, wherein the mechanism provides for the identified entity of the comparing to express the privacy preferences to suppress a requester of the comparing from the receiving a visual query result that is associated with the identified entity, for the requester not being logged into an online account, or the requester not being in a social group of the identified entity, and further, the mechanism allows the individual subject of the query to express a preference for receiving an alert indicative of a use of the facial image of the identified entity in the second database records, and an online delivery address and frequency preference for the comparing and for receiving the alert; and

initiate advising the identified entity, corresponding to the at least one of the first database records, if the comparing yields a probable match.

25. The apparatus of claim 24, further comprising a plurality of distinct software modules, each of the distinct software modules being embodied in a non-transitory manner on a tangible computer-readable recordable storage medium, and wherein the distinct software modules comprise a first database module, a second database module, and a search system module;

wherein;

the at least one processor is operative to obtain access to the first database by executing the first database module;

the at least one processor is operative to obtain access to the second database by executing the second database module;

the at least one processor is operative to compare by executing the search system module; and

the at least one processor is operative to initiate advising the given one of the entities by executing the search system module.

26. An apparatus for alerting an entity to use of an image, the apparatus comprising:

means for obtaining access to a first database having a plurality of first database records, each of the first database records comprising at least;

an identification of a corresponding one of a plurality of entities; and

a plurality of features of a plurality of images associated with the entities;

means for obtaining access to a second database having a plurality of second database records, each of the second database records comprising at least;

an identification of a corresponding one of a plurality of generalized web locations; and

a plurality of features of a plurality of images associated with the generalized web locations, the plurality of features being extracted from images obtained by crawling the generalized web locations;

means for, for at least one of the first database records, comparing corresponding ones of the features of a corresponding one of the images associated with a corresponding one of the entities to the plurality of features in the second database records to determine an identified entity of the plurality of entities;

means for obtaining, from the identified entity, via a mechanism for expressing privacy preferences, at least one privacy preference comprising a visual query preference of the identified entity, wherein the mechanism provides for the identified entity of the comparing to express the privacy preferences to suppress a requester of the comparing from the receiving a visual query result that is associated with the identified entity, for the requester not being logged into an online account, or the requester not being in a social group of the identified entity, and further, the mechanism allows the individual subject of the query to express a preference for receiving an alert indicative of a use of the facial image of the identified entity in the second database records, and an online delivery address and frequency preference for the comparing and for receiving the alert; and

means for initiating advising the identified entity, corresponding to the at least one of the first database records, if the comparing yields a probable match.